Semantic segmentation method, system and equipment and storage medium

Abstract

The invention provides a semantic segmentation method, system and device and a storage medium. The semantic segmentation method comprises the following steps: respectively extracting voxel features and image features based on initial point cloud data and image data of a target scene; obtaining a first point-by-point feature and a second point-by-point feature according to the voxel feature and the image feature so as to further obtain a geometric fusion feature and a semantic fusion feature; modeling according to the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model; and inputting to-be-segmented target point cloud data into the semantic segmentation model so as to perform semantic segmentation on a target scene. According to the invention, the continuity and integrity of the features are improved; and spatial structure information can be reserved, the semantic understanding depth is enhanced, and global context features can be captured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a semantic segmentation method, system, device and storage medium. Background Art

[0002] Semantic segmentation is one of the important tasks in the environmental perception link of autonomous driving technology. It aims to understand the scenes (pictures, point clouds, etc.) obtained by sensors and mark the semantic information of different areas. It is the basis of visual tasks such as target recognition and classification.

[0003] An existing semantic segmentation method loses some features due to discontinuous projection views; another method has limited local feature extraction capabilities and cannot capture global context features well for point clouds of large scenes. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a semantic segmentation method, system, device and storage medium that can ensure the integrity and continuity of feature extraction and effectively capture global context features.

[0005] Specifically, the present application provides a semantic segmentation method, comprising the following steps:

[0006] Initial point cloud data and image data of a target scene are acquired, voxel features are extracted based on the initial point cloud data, and image features are extracted based on the image data.

[0007] The voxel features and the image features are mapped to the initial point cloud data to obtain a first point-by-point feature and a second point-by-point feature.

[0008] A geometric fusion process is performed based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and a semantic fusion process is performed based on the voxel feature and the image feature to obtain a semantic fusion feature.

[0009] Modeling is performed based on the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model.

[0010] And, obtaining target point cloud data to be segmented, and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0011] In the above technical scheme, the continuity and integrity of features are improved by comprehensively utilizing image data and point cloud data; by geometrically fusing and semantically fusing voxel features and image features, not only the spatial structure information is retained, but also the depth of semantic understanding is enhanced. This dual fusion can better capture features in complex environments and help capture global context features.

[0012] In addition, since the model is built based on the fusion of two feature data, it can better ensure the semantic consistency between point cloud data and image data when performing semantic segmentation, and reduce the errors caused by different data sources.

[0013] Furthermore, the extracting of voxel features includes:

[0014] The initial point cloud data is segmented based on the quantization step size, and a plurality of groups of non-empty voxels are obtained according to the segmented initial point cloud data.

[0015] Furthermore, multiple groups of non-empty voxels are integrated into a group of point cloud spaces for rasterization, and voxel features of each group of non-empty voxels are extracted by superimposing sparse convolution and sparse deconvolution.

[0016] In the above technical scheme, by segmenting the initial point cloud data based on the quantization step size, the point cloud data can be processed in a refined manner, which can effectively reduce the complexity of the data and make subsequent feature extraction more efficient and accurate; extracting multiple groups of non-empty voxels can focus on important spatial areas, which helps to reduce the waste of computing resources and improve the processing speed. The selection of non-empty voxels enables the model to focus on areas containing actual information, increasing the effectiveness of feature extraction; integrating multiple groups of non-empty voxels into a group of point cloud spaces for rasterization helps to unify the data format and structure for subsequent operations; by superimposing sparse convolution and sparse deconvolution to extract voxel features, local and global features can be better captured while maintaining computational efficiency, while retaining spatial information, so that the model can more effectively understand and segment targets when facing complex scenes.

[0017] The design of this method makes it suitable for processing large-scale point cloud data. The characteristics of sparse convolution allow the model to remain efficient under large-scale data, thereby improving the practicality and applicability of the system.

[0018] Furthermore, the extracting of image features includes:

[0019] The image data is subjected to nonlinear projection and downsampling operations through an image backbone to extract image features.

[0020] In the above technical solution, when the image backbone performs nonlinear projection and downsampling operations, it actually extracts features through different levels of convolution layers and pooling layers. This multi-level feature extraction can capture a variety of features from low-level to high-level, including edges, textures, shapes, and object contours; nonlinear projection can introduce complex nonlinear feature mapping relationships, making the features extracted from the image more expressive; downsampling operations can effectively reduce the size of the feature map, thereby reducing the amount of calculation and memory requirements, and improving the efficiency of feature extraction.

[0021] Furthermore, despite the downsampling operation, the design in the image backbone network usually maintains important spatial structure information by using an appropriate step size and retaining a certain spatial resolution, which helps to better maintain spatial continuity and consistency when performing semantic segmentation.

[0022] Further, the obtaining of the first point-by-point feature and the second point-by-point feature includes:

[0023] The voxel feature is used as the first point-by-point feature.

[0024] And, obtaining point-by-point image coordinates, and based on the initial point cloud data, performing bilinear interpolation processing on the image features through the point-by-point image coordinates to obtain second point-by-point features.

[0025] In the above technical solution, bilinear interpolation processing is performed on image features based on point-by-point image coordinates, so that image features can be selectively extracted at different scales and resolutions. This interpolation method can effectively reduce the errors caused by spatial discretization, thereby improving the feature representation ability; and through high-quality point-by-point feature acquisition, the semantic segmentation model can more accurately understand the characteristics of each point and the surrounding environment, thereby improving the segmentation accuracy.

[0026] In addition, bilinear interpolation can also effectively process feature information at different scales and adapt to various transformations in the scene, such as changes in perspective and object size, thereby enhancing the adaptability of the model.

[0027] Furthermore, the obtaining of geometric fusion features includes:

[0028] The first point-by-point feature and the second point-by-point feature are concatenated to obtain a geometric fusion feature.

[0029] In the above technical solution, by splicing the first point-by-point feature (voxel feature) and the second point-by-point feature (image feature), information from different data sources can be fully integrated, so that the model can more comprehensively understand the geometric characteristics and semantic information of the target scene.

[0030] Furthermore, the semantic fusion feature includes at least a first semantic feature and a second semantic feature; and obtaining the semantic fusion feature includes:

[0031] First intermediate segmentation features are predicted based on the voxel features, and each first intermediate segmentation feature is normalized to obtain a first semantic feature.

[0032] And, predicting a second intermediate segmentation feature based on the image feature, and performing a rearrangement operation and a normalization process on the second intermediate segmentation feature to obtain a second semantic feature.

[0033] In the above technical solution, whether it is the normalization processing of the first intermediate segmentation feature or the second intermediate segmentation feature, it can effectively reduce the difference between the features, making the features more stable and consistent in subsequent model processing, which helps to improve the training efficiency and generalization ability of the model; the first and second semantic features generated based on voxels and image features respectively can provide rich contextual information; the second intermediate segmentation feature is rearranged so that the image features can be integrated into the segmentation task in the most efficient way.

[0034] Furthermore, obtaining the semantic segmentation model includes:

[0035] The geometric fusion feature, the first semantic feature and the second semantic feature are projected into a multi-dimensional space respectively to obtain point-by-point projection features, and the point-by-point projection features are spliced ​​to obtain typical features.

[0036] A model is built based on the semantic relationship of the typical features through multi-head self-attention, and the multi-head self-attention is decomposed into single-head self-attention to obtain semantic embedding features.

[0037] Secondary modeling is performed based on the semantic relationship between the point-by-point projection features and the semantic embedding features through multi-head cross attention to obtain a semantic segmentation model.

[0038] In the above technical solution, the mapping of multi-dimensional space enables different features to more effectively capture and express complex semantic relationships in higher-dimensional space; the generation of typical features enables the model to understand all aspects of the scene more comprehensively; the multi-head self-attention mechanism can observe the relationship between different features in parallel, improving the model's contextual understanding ability; decomposing the multi-head self-attention into single-head self-attention can not only reduce the computational complexity, but also improve the model's reasoning speed and efficiency. Single-head self-attention is more suitable for real-time application scenarios while maintaining computational accuracy; multi-head cross attention can process multi-source features in parallel, improving the accuracy of semantic segmentation.

[0039] Furthermore, the entire modeling process undergoes multiple hierarchical processing, including multi-dimensional space projection, splicing processing, self-attention and cross-attention modeling, to form a multi-level semantic understanding system, which can deeply understand the semantic structure of complex scenes and significantly improve the segmentation quality.

[0040] Based on the same concept, the present application also provides a semantic segmentation system, which adopts the semantic segmentation method and includes:

[0041] The extraction module is used to obtain initial point cloud data and image data of the target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data.

[0042] A mapping module is used to map the voxel feature and the image feature to the initial point cloud data to obtain a first point-by-point feature and a second point-by-point feature.

[0043] A fusion module is used to perform geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and to perform semantic fusion processing based on the voxel feature and the image feature to obtain a semantic fusion feature.

[0044] The modeling module is used to perform modeling based on the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model.

[0045] And a segmentation module is used to obtain target point cloud data to be segmented, and input the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0046] In the above technical solution, by combining image data and point cloud data, the continuity and integrity of features are significantly improved. Specifically, by geometrically fusing voxel features and image features and semantically fusing them, both spatial structure information is retained and the depth of semantic understanding is enhanced. This dual fusion mechanism can more effectively capture features in complex environments, especially global context features, thereby improving the model's ability to understand complex scenes.

[0047] In addition, since the model is built based on the fusion of two types of feature data, it can better ensure the semantic consistency between point cloud data and image data during semantic segmentation, and reduce the errors caused by differences in data sources. This approach not only improves the accuracy of the segmentation results, but also enhances the robustness and adaptability of the model in different environments.

[0048] Based on the same concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the semantic segmentation method when executing the computer program.

[0049] Based on the same concept, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the semantic segmentation method is implemented.

[0050] Compared with the prior art, the beneficial effects of this application are:

[0051] This application makes comprehensive use of image data and point cloud data to improve the continuity and integrity of features; it can effectively retain spatial structural information, enhance the depth of semantic understanding, and thus achieve effective acquisition of global context features. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the semantic segmentation method described in an embodiment of the present application.

[0053] Figure 2 This is a framework diagram of the semantic segmentation system described in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following is a further detailed description of a semantic segmentation method, system, device and storage medium of the present application in conjunction with specific embodiments and drawings.

[0055] See also Figure 1 , the embodiment of the present application provides a semantic segmentation method, comprising the following steps:

[0056] Initial point cloud data and image data of a target scene are acquired, voxel features are extracted based on the initial point cloud data, and image features are extracted based on the image data.

[0057] The nuScenes dataset is divided into 28,130 training samples, 6,019 validation samples, and 6,008 test samples [3,11]. Each nuScenes sample contains a relatively sparse 32-bundle point cloud (i.e., the initial point cloud data) and an RGB image (i.e., the image data) captured by 6 cameras: front, front left, front right, back, back left, and back right.

[0058] It should be noted that the above numerical values ​​are only exemplary, and those skilled in the art can set and select the numerical values ​​according to actual needs.

[0059] In some embodiments, extracting voxel features includes:

[0060] The initial point cloud data is segmented based on the quantization step size, and a plurality of groups of non-empty voxels are obtained according to the segmented initial point cloud data.

[0061] In some embodiments, a general sparse 3D U_Net9,35] is used for voxel-based LiDAR feature extraction, and the initial point cloud data is radar point cloud data. The input radar point cloud data is first segmented by a quantization step size d, and then grouped into non-empty voxels by averaging the point-by-point initial features of local points within the voxel.

[0062] The 3D U_Net9,35] is a deep learning network architecture for 3D image segmentation, which is an extension of the traditional U-Net; its convolution, pooling and upsampling operations are all performed in three-dimensional space, which enables it to effectively utilize the spatial information of three-dimensional images.

[0063] Among them, these non-empty voxels Vin∈Rwexl constitute the sparse tensor input of the point cloud backbone, which is used to learn to express the voxel features V∈RNvaxelXCvoxel.

[0064] Vin∈RwexlX means that the input data Vin is a real tensor with a shape of (width, length); V∈RNvaxelXCvoxel: means that the voxel feature V is a real tensor with a shape of (Nvaxel, Cvoxel), where Nvaxel is the number of voxels and Cvoxel is the feature dimension of each voxel.

[0065] Furthermore, multiple groups of non-empty voxels are integrated into a group of point cloud spaces for rasterization, and voxel features of each group of non-empty voxels are extracted by superimposing sparse convolution and sparse deconvolution.

[0066] Further, in some embodiments, due to the superiority of 3D voxels, the entire point cloud space is first rasterized to achieve data structuring, and the resolution of these spatial voxel grids is L×W×H. The voxel features of each non-empty voxel are extracted by superimposing sparse convolution and sparse deconvolution, where the initial feature of each voxel is simply calculated as the average value of the coordinates of the points within each voxel in the LiDAR coordinate system. By calculating the mean of the points within each non-empty voxel grid, a new point cloud is formed with point features (i.e., voxel features), because the size of the voxel is much smaller than the entire 3D space (e.g., 5cm×5cm×10cm in the method), the voxelized point cloud can not only be processed by a more efficient sparse convolution-based backbone, but also is approximately equivalent to the original point cloud in three-dimensional target detection.

[0067] Among them, a 3D sparse convolution (i.e., the sparse convolution) and 3D sparse deconvolution (i.e., the sparse deconvolution) are performed on the obtained sparse voxels using a voxel encoding and decoding architecture similar to U-net; the downsampling process is composed of four sparse convolution modules with dimensions of 16-32-64-64, and each sparse convolution layer uses a convolution kernel size of 3*3*3, a stride of 2, and a downsampling factor of 8. Two sub-population convolution layers with a convolution kernel size of 3*3*3 and a stride of 1 are stacked behind each sparse convolution layer.

[0068] In some embodiments, extracting image features includes:

[0069] The image data is subjected to nonlinear projection and downsampling operations through an image backbone to extract image features.

[0070] In some embodiments, in order to enhance LiDAR features with jointly optimized camera features, a trainable image backbone (HRNet-w481441 by default) is used to nonlinearly project image data Xin as X∈RNom×Cimg×HxW, where RNom represents the number of feature maps, HxW represents the downsampling shape (including height H and width W), and the channel Cimg is added. Point cloud and image backbones can be flexibly selected from a variety of mature networks. The two backbones in the entire 3D segmentation model are jointly optimized to learn relevant semantic feature representations from different modalities. The expressive voxel features V and image features X ensure the fusion of cross-modal features.

[0071] Among them, HRNet-w48 is a high-resolution network (High-Resolution Network) composed of multiple branches. It can extract features at different resolutions and is very suitable for tasks such as image segmentation. The trainable parameters of this backbone network can be adjusted and optimized through back propagation.

[0072] The "non-linear projection" here refers to processing the input image through the backbone network to generate new feature representations. This processing usually involves multi-layer convolution, activation functions, and pooling operations.

[0073] Furthermore, after the voxel features and the image features are extracted, the voxel features and the image features are mapped to the initial point cloud data to obtain the first point-by-point features and the second point-by-point features.

[0074] In some embodiments, the point-centered segmentation excitation is performed by using the geometric association of lidar and multiple cameras to map voxel features V and image features X to points in Pin (i.e., the initial point cloud data) as point-by-point lidar features (i.e., the first point-by-point features) Flidar∈RNpoint×Cvoxel and point-by-point camera features (i.e., the second point-by-point features) Fcam∈RNpoint×Cimg.

[0075] Among them, Npoint represents the number of features.

[0076] In some embodiments, obtaining the first point-by-point feature and the second point-by-point feature includes:

[0077] The voxel feature is used as the first point-by-point feature.

[0078] And, obtaining point-by-point image coordinates, and based on the initial point cloud data, performing bilinear interpolation processing on the image features through the point-by-point image coordinates to obtain second point-by-point features.

[0079] In some embodiments, the voxel feature V is used as the point-by-point lidar feature Flidar=[flidar,i]i=Npointi=1∈RNpoint×Cvoxel to enhance point-by-point fusion and segmentation.

[0080] Further, given a point (i.e., any point in the initial point cloud data), its point features are interpolated from the three nearest voxels; for the i-th point (xi, yi, zi) in Flidar with lidar function flidar,i, bilinear interpolation I is performed on the image features X[ci]∈RCimg×H×W within the ci-th local camera using the known point-by-point image coordinates (ci, ui, vi), as 1. Zeros are temporarily padded to those points outside the camera FOV, which allows each point to be decorated with Cimg-dimensional features fcam,i. In order to distinguish external points, the binary mask B of the Npoint elements is set to 0 for them and 1 otherwise.

[0081] in:

[0082]

[0083] Furthermore, after obtaining the first point-by-point feature and the second point-by-point feature, geometric fusion processing is performed based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and semantic fusion processing is performed based on the voxel feature and the image feature to obtain a semantic fusion feature.

[0084] In some embodiments, obtaining the geometric fusion feature includes:

[0085] The first point-by-point feature and the second point-by-point feature are concatenated to obtain a geometric fusion feature.

[0086] In some embodiments, the point-by-point lidar features Flidar and the point-by-point camera features Fcam are projected into Cint dimensions by using a fully connected layer, and are concatenated together. So that learnable fusion can be subsequently performed using another MLP whose Cgfused output channel fgfused,i = MLP(Flidar(flidar,i) Fcam(fcam,i)) and Fgfused=[fgfused,i]i=Npointi=1∈RNpoint×Cgfused. GF-Phase combines the multimodal feature Fgfused mainly with geometric clues, which is called the geometry-based feature fusion stage.

[0087] In some embodiments, the semantic fusion feature includes at least a first semantic feature and a second semantic feature; and obtaining the semantic fusion feature includes:

[0088] First intermediate segmentation features are predicted based on the voxel features, and each first intermediate segmentation feature is normalized to obtain a first semantic feature.

[0089] In some embodiments, the LiDAR semantic feature aggregation module (LiDAR SFAM) starts from the voxel features V and ends with the LiDAR semantic embedding in the aggregation NclsElidar. For the Ncls category, assume Elidar∈RNcls×Cvoxel and a distribution matrix Dlidar∈(0,1)Ncls×Nvoxel; where (0,1) indicates that the value range of the element is between 0 and 1, Ncls indicates the number of categories, and Nvoxel indicates the number of voxels.

[0090] D[j,i] being closer to 1 indicates that the i-th voxel is more likely to belong to the j-th category than other voxels, so it contributes more to describing this category in Elidar[j]. Otherwise, the i-th voxel should contribute less to Elidar[j]. Elidar can be calculated by matrix product Elidar = DlidarV. Two more steps are required to calculate Dlidar. First, the mlp-based auxiliary voxel segmentation head Hvoxel (Hvoxel represents the height of each voxel) is used to predict the intermediate segmentation D′lidar∈RNvoxel×Ncls on the voxel feature V (that is, predict the first intermediate segmentation feature). Then, a spatial softmax is performed between voxels to normalize the values ​​of each category to (0,1) (that is, each first intermediate segmentation feature is normalized).

[0091] in:

[0092]

[0093] And, predicting a second intermediate segmentation feature based on the image feature, and performing a rearrangement operation and a normalization process on the second intermediate segmentation feature to obtain a second semantic feature.

[0094] In some embodiments, for more accurate Dlidar, the D′ lidar is explicitly guided by using voxel segmentation supervision.

[0095] The camera semantic feature aggregation module (Camera sfm) similarly aggregates Ecam∈RNcls×Cimg from the image feature map X∈RNcam×Cimg×H×W using another distribution matrix Dcam∈(0,1)Ncls×Npixel and another auxiliary image segmentation head Himg implemented by a simple FCN head. The difference is that the camera sfm designs the image feature map of all Npixel pixels of the Ncam local camera, where Npixel is Ncam×H×W. The Himg head predicts the intermediate segmentation D′img∈RNcam×Ncls×H×W by Himg(X).

[0096] Among them, Npixel represents the number of pixels; RNcam represents the number of images or cameras in the data batch; Cimg represents the number of channels of the image; H×W represents the height and width of the image.

[0097] It should be noted that through the transposition and reshape operations, D′img and X are rearranged as D′cam∈RNcls×Npixel and X′∈RNpixel×Cimg. The final camera semantic embedding Ecam is the matrix product of Dcam and X′, where the normalized Dcam is the spatial softmax applied on D′cam.

[0098] in:

[0099] D′ img =H img (X)

[0100]

[0101] E cam =D cam X

[0102] Furthermore, after obtaining the geometric fusion features and the semantic fusion features, modeling is performed according to the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model.

[0103] In some embodiments, obtaining a semantic segmentation model includes:

[0104] The geometric fusion feature, the first semantic feature and the second semantic feature are projected into a multi-dimensional space respectively to obtain point-by-point projection features, and the point-by-point projection features are spliced ​​to obtain typical features.

[0105] In some embodiments, given the input of the point-by-point geometric fusion feature Fgfused, the category-by-category semantic embedding Elidar (i.e., the first semantic feature) and Ecam (i.e., the second semantic feature), SFFM first projects them into the Csfused dimensional space Fproj=Proj1(Fgfused), and E=Proj2(Elidar) respectively. Proj3 (Ecam). Before stacking K grayscale patches in SFFM, the projected multimodal semantic embeddings are concatenated together to form E∈R2Ncls×Csfused, which can be viewed as a dictionary describing the typical features of each category from the perspective of lidar and camera.

[0106] A model is built based on the semantic relationship of the typical features through multi-head self-attention, and the multi-head self-attention is decomposed into single-head self-attention to obtain semantic embedding features.

[0107] Secondary modeling is performed based on the semantic relationship between the point-by-point projection features and the semantic embedding features through multi-head cross attention to obtain a semantic segmentation model.

[0108] In some embodiments, in each gray block, multi-head self-attention (MHSA) is used to model the semantic relationship between all 2Ncls classification semantic embeddings, where Norm represents the LayerNorm operation. Let Cshsa be Csfused / NH, MHSA can be decomposed into NH single-head self-attention (SHSA) operation, Eh∈R2Ncls×Cshsa is:

[0109] E mhsa =Norm(E+MHSA(E)),

[0110]

[0111] After updating E to Emhsa, the point-by-point projection feature Fproj and the semantic embedding feature Emhsa are further fused, and the multi-head cross attention (MHCA) is used for semantic relationship modeling, and the feedforward network (FFN) is used for feature embedding; where:

[0112] F sfused =Norm(F mhca +FFN(F mhca )),

[0113] F mhca =Norm(F proj +MHCA(F proj ,E mhsa ,E mhsa )).

[0114] It should be noted that Ncls is preferably 17, and those skilled in the art may also set it to other values ​​according to actual application requirements.

[0115] Furthermore, after obtaining the semantic segmentation model, the target point cloud data to be segmented is obtained, and the target point cloud data is input into the semantic segmentation model to perform semantic segmentation on the target scene.

[0116] Further, in some embodiments, see Figure 2 The present application also provides a semantic segmentation system, wherein the semantic segmentation system adopts the semantic segmentation method and comprises:

[0117] The extraction module is used to obtain initial point cloud data and image data of the target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data.

[0118] A mapping module is used to map the voxel feature and the image feature to the initial point cloud data to obtain a first point-by-point feature and a second point-by-point feature.

[0119] A fusion module is used to perform geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and to perform semantic fusion processing based on the voxel feature and the image feature to obtain a semantic fusion feature.

[0120] The modeling module is used to perform modeling based on the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model.

[0121] And a segmentation module is used to obtain target point cloud data to be segmented, and input the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

[0122] Furthermore, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the semantic segmentation method when executing the computer program.

[0123] In some embodiments, the memory and the processor are interconnected via a bus; the processor may be one or more CPUs. When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor is used to control various functional modules of the electronic device and process signals.

[0124] The memory includes, but is not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), and CD-ROM (Compact Disc Read-Only Memory). The memory is used to store computer programs, operating systems, various applications and data, such as storing computer programs for implementing the semantic segmentation method.

[0125] Furthermore, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the semantic segmentation method is implemented.

[0126] In some embodiments, the storage medium stores several computer programs for causing a device to execute all or part of the steps of the method described in various embodiments of the present application.

[0127] The medium may include a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and other media that can store program codes.

[0128] In summary, the present application provides a semantic segmentation method, system, device and storage medium; firstly, voxel features and image features are extracted based on the initial point cloud data and image data of the target scene; then, the first point-by-point features and the second point-by-point features are obtained according to the voxel features and the image features to further obtain geometric fusion features and semantic fusion features; then, modeling is performed according to the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model; and then the target point cloud data to be segmented is input into the semantic segmentation model to perform semantic segmentation on the target scene. The present application improves the continuity and integrity of features; and not only can it retain spatial structural information, but it also enhances the depth of semantic understanding, which helps to capture global context features.

[0129] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0130] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0132] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some modules according to the embodiments of the present application. The application can also be implemented as a device program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0133] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] Although the description of the present application is carried out in conjunction with the above specific embodiments, it is obvious that those skilled in the art can make many substitutions, modifications and changes according to the above content. Therefore, all such substitutions, improvements and changes are included in the spirit and scope of the appended claims.

Claims

1. A semantic segmentation method, characterized in that: The following steps are involved: Acquire initial point cloud data and image data of the target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data; Mapping the voxel features and the image features to the initial point cloud data to obtain a first point-by-point feature and a second point-by-point feature; Performing geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and performing semantic fusion processing based on the voxel feature and the image feature to obtain a semantic fusion feature; Modeling is performed based on the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model; And, obtaining target point cloud data to be segmented, and inputting the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

2. The semantic segmentation method according to claim 1, characterized in that: The extracting voxel features comprises: Segmenting the initial point cloud data based on a quantization step size, and obtaining a plurality of groups of non-empty voxels according to the segmented initial point cloud data; Furthermore, multiple groups of non-empty voxels are integrated into a group of point cloud spaces for rasterization, and voxel features of each group of non-empty voxels are extracted by superimposing sparse convolution and sparse deconvolution.

3. The semantic segmentation method according to claim 2, characterized in that: The extracting of image features comprises: The image data is subjected to nonlinear projection and downsampling operations through an image backbone to extract image features.

4. The semantic segmentation method according to claim 3, characterized in that: The obtaining of the first point-by-point feature and the second point-by-point feature comprises: Using the voxel feature as a first point-by-point feature; And, obtaining point-by-point image coordinates, and based on the initial point cloud data, performing bilinear interpolation processing on the image features through the point-by-point image coordinates to obtain second point-by-point features.

5. The semantic segmentation method according to claim 4, characterized in that: The obtaining of geometric fusion features comprises: The first point-by-point feature and the second point-by-point feature are concatenated to obtain a geometric fusion feature.

6. The semantic segmentation method according to claim 5, characterized in that: The semantic fusion feature includes at least a first semantic feature and a second semantic feature; and obtaining the semantic fusion feature includes: Predicting first intermediate segmentation features based on the voxel features, and normalizing each first intermediate segmentation feature to obtain a first semantic feature; And, predicting a second intermediate segmentation feature based on the image feature, and performing a rearrangement operation and a normalization process on the second intermediate segmentation feature to obtain a second semantic feature.

7. The semantic segmentation method according to claim 6, characterized in that: The obtaining of the semantic segmentation model comprises: Projecting the geometric fusion feature, the first semantic feature, and the second semantic feature into a multidimensional space respectively to obtain point-by-point projection features, and performing splicing processing on the point-by-point projection features to obtain typical features; Modeling is performed once based on the semantic relationship of the typical features through multi-head self-attention, and the multi-head self-attention is decomposed into single-head self-attention to obtain semantic embedding features; Secondary modeling is performed based on the semantic relationship between the point-by-point projection features and the semantic embedding features through multi-head cross attention to obtain a semantic segmentation model.

8. A semantic segmentation system, characterized in that: The semantic segmentation system adopts the semantic segmentation method according to any one of claims 1 to 7, including: An extraction module, used to obtain initial point cloud data and image data of a target scene, extract voxel features based on the initial point cloud data, and extract image features based on the image data; A mapping module, used for mapping the voxel feature and the image feature to the initial point cloud data to obtain a first point-by-point feature and a second point-by-point feature; A fusion module, configured to perform geometric fusion processing based on the first point-by-point feature and the second point-by-point feature to obtain a geometric fusion feature, and perform semantic fusion processing based on the voxel feature and the image feature to obtain a semantic fusion feature; A modeling module, used for modeling according to the geometric fusion features and the semantic fusion features to obtain a semantic segmentation model; And a segmentation module is used to obtain target point cloud data to be segmented, and input the target point cloud data into the semantic segmentation model to perform semantic segmentation on the target scene.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the semantic segmentation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the semantic segmentation method according to any one of claims 1 to 7 is implemented.

Images