Three-dimensional point cloud segmentation method and system based on bidirectional fusion of point cloud and aerial view

Abstract

The invention provides a three-dimensional point cloud segmentation method and system based on bidirectional fusion of a point cloud and an aerial view, and belongs to the technical field of three-dimensional scene environment perception, and the method comprises the steps: obtaining to-be-segmented original point cloud data; processing the obtained original point cloud data by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network and a fusion segmentation head. According to the method, a bidirectional fusion mechanism of the point cloud and the aerial view is provided, through interactive fusion of the point cloud and the BEV features in multiple stages, the expression ability of the model for local and global features is improved, the problems of fuzzy semantic boundary, difficulty in small target recognition and the like are remarkably improved, and the segmentation precision is obviously superior to that of an existing BEV method. Compared with a voxel method and a multi-representation fusion method, the method has the advantages that the weight of a calculation graph is kept light in structural design, compared with the voxel method and the multi-representation fusion method, the method has remarkable advantages in the aspects of parameter quantity and reasoning speed, precision and real-time performance are better considered, and the method is suitable for scenes with extremely high efficiency requirements such as automatic driving.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional scene environment perception technology, and in particular to a three-dimensional point cloud segmentation method and system based on bidirectional fusion of point cloud and bird's-eye view. Background Art

[0002] 3D point cloud segmentation is a core task in autonomous driving, robotics, and other safety-critical applications. It aims to provide fine-grained, point-level semantic understanding for environmental perception. In recent years, with the rapid development of LiDAR and 3D perception technologies, point cloud segmentation has made significant progress. However, achieving efficient and high-precision point cloud segmentation in resource-constrained real-world driving scenarios remains challenging. Therefore, designing a 3D point cloud segmentation system that balances computational efficiency and accuracy has become a key research topic.

[0003] Existing 3D point cloud segmentation methods are mainly based on the following data representation methods. First, the original point cloud based method, such as Figure 1 As shown in [1], these methods directly process input point data and extract local and global features of each point using a multi-layer perceptron (MLP). Methods such as PointNet and its variant, Randla-net, employ point-independent feature extraction and a global pooling mechanism, further enhancing local semantic expression through neighborhood queries and feature aggregation. These methods typically preserve the geometric structure of point clouds and possess strong feature modeling capabilities. However, their feature extraction relies on computationally complex neighborhood queries and graph construction, making them difficult to meet real-time requirements in large-scale scenarios.

[0004] Second, voxel-based methods, such as Figure 2 As shown in the figure, it divides the point cloud into equally spaced three-dimensional grid units (Voxel), converting the point cloud from an irregular representation into a regular tensor, which can be directly processed by a three-dimensional convolutional neural network. In recent years, the introduction of sparse convolution has made this type of method more efficient in processing sparse voxel data. Representative works such as MinkUNet and Cylinder3D extract spatial features of different scales through multi-layer sparse convolution and perform voxel-by-voxel semantic classification. This type of method significantly improves the computing speed while ensuring a certain level of accuracy, especially in medium and low resolution scenarios. However, in applications pursuing high precision, the high-resolution setting required for voxelization still brings a significant computational burden.

[0005] Third, projection-based methods, such as Figure 3As shown in the figure, it maps the three-dimensional point cloud into a two-dimensional image form and introduces a two-dimensional semantic segmentation model for efficient processing. Common representations include the bird's-eye view (BEV) and the range view. The bird's-eye view projects the point cloud to the ground plane, capturing the spatial layout and geometric structure information of the scene, which facilitates the model to understand and predict in a large range of scenes. Representative works include PolarNet; the range map generates an image representation through depth and angle information, which is suitable for modeling the local structure and depth relationship of the point cloud. Representative works include CENet. This type of method is convenient for end-to-end training and reasoning with the help of the existing 2D network architecture, and has good real-time processing capabilities. However, problems such as incompleteness caused by projection, severe projection distortion, and lack of local semantics need to be solved, resulting in poor segmentation accuracy of this type of method.

[0006] Finally, methods based on multi-representation fusion, such as Figure 4 As shown in [1], it uses a combination of representations, such as points, voxels, and projections, to achieve joint modeling of multi-scale and multi-view semantic information through parallel or alternating feature extraction and fusion modules. A typical method is RPVNet, which extracts features from the original point cloud while converting them into voxels or projections for auxiliary feature modeling. It then performs feature alignment and fusion in subsequent stages to improve overall segmentation accuracy. This type of method combines the structural advantages of various representations and has achieved excellent performance on multiple public benchmark datasets in recent years. However, the fusion and processing of multi-representation features inevitably introduces higher computational costs, resulting in high computational overhead.

[0007] Among various 3D point cloud segmentation methods, projection-based methods have become a viable approach for achieving real-time 3D point cloud segmentation due to their ability to fully leverage mature 2D convolutional networks and their high computational efficiency. Bird's-eye-view representations, in particular, are gaining increasing attention due to their advantages in capturing the overall spatial structure and geometric layout of a scene, facilitating the understanding and modeling of large-scale scenes.

[0008] However, the bird's-eye view method also has significant limitations. For one thing, information compression inevitably occurs during the projection of point clouds into two-dimensional images, resulting in a loss of contextual information about spatial neighborhoods. Furthermore, some key points may be overlooked during the projection process, resulting in incomplete representation. These issues reduce the model's ability to perceive small objects and boundary regions, often requiring the introduction of additional post-processing modules to complement these, but this has limited effectiveness and limits the performance ceiling of the bird's-eye view method. Summary of the Invention

[0009] The application aims to provide a three-dimensional point cloud segmentation method and system based on bidirectional fusion of point cloud and bird's eye view, which realizes deep fusion of point cloud fine-grained features and BEV global structure information by designing a BEV-point cloud joint encoder and a bidirectional fusion attention module, and improves the segmentation accuracy and boundary perception ability of the model by introducing a fusion segmentation head and a multi-loss supervision mechanism. Experiments on multiple public data sets prove that the segmentation accuracy is greatly improved while ensuring efficient inference speed, and the method is suitable for real-time automatic driving perception and intelligent monitoring system, and solves at least one technical problem in the above background art.

[0010] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0011] In a first aspect, the application provides a three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view, comprising:

[0012] Obtaining original point cloud data to be segmented;

[0013] Processing the obtained original point cloud data by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them to the BEV space, generate preliminary BEV features after processing by a two-dimensional convolution layer, then project them back to the point cloud space, fuse them with the initial features of the point cloud, and further fuse the fused point cloud features with the BEV features obtained by the previous processing to form the preliminary fusion representation of the point cloud and BEV; the backbone network takes the preliminary fusion representation of the point cloud and BEV as input, adopts a structure of alternately stacking multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules, realizes bidirectional interaction enhancement of point cloud and BEV features layer by layer, and outputs point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

[0014] As a further limitation of the first aspect of the application, the calculation formula in the encoder module is as follows:

[0015]

[0016] wherein, For the original point cloud, MLP(·) represents a fully connected layer with batch normalization and Relu activation function; Conv(·) represents a 3×3 convolutional layer with batch normalization and HSwish activation function; P2B(·) represents the projection of point cloud features to BEV representation, where multiple points may be mapped to the same pixel position; B2P(·) represents the back projection of BEV features to point representation, where points falling into the same pixel position share the same features; Cat(·) represents tensor weighting operation.

[0017] As a further limitation of the first aspect of the present invention, the backbone network is composed of multiple layers of alternating two-dimensional residual convolution blocks and bidirectional fusion attention modules, designed to achieve multi-level point cloud and BEV feature fusion. The initial fusion of the point cloud and BEV is first input into the backbone network. In each layer, the BEV features are first processed by the two-dimensional residual convolution blocks to extract the semantic information of the layer. They are then input into the bidirectional fusion attention module together with the corresponding point cloud features to achieve interactive fusion of the point cloud and BEV.

[0018] In the bidirectional fusion attention module, the BEV features of the current layer are first back-projected into the point cloud space via spatial mapping and concatenated with the point cloud features of the current layer in the channel dimension to form a cross-modal joint representation. The concatenated features are then aligned using a multi-layer perceptron (MLP), which outputs enhanced point cloud features to improve their ability to model global contextual information. These enhanced point cloud features are then projected back into the BEV space, concatenated with the BEV features of the current layer, and fed into a two-dimensional convolutional network for fusion processing to enhance the local structural representation of the BEV features.

[0019] Furthermore, a multi-level feature attention module (HFA) is introduced to perform fine-grained enhancement on the fused BEV features. This module consists of three branches connected in series: channel attention, spatial attention, and semantic attention, which respectively model channel response strength, key spatial regions, and cross-layer semantic consistency. These three attention types act on different dimensions of the BEV features through a weighted fusion mechanism, enhancing key semantic information and suppressing redundant information. Finally, the enhanced BEV features are element-wise added to the output of the original convolutional path of the backbone network to achieve a residual connection.

[0020] The point cloud features and BEV features output by this layer will be used as the input of the bidirectional fusion attention module and the two-dimensional residual convolution block of the next layer respectively, realizing the layer-by-layer fusion and feature progression of the point cloud and BEV in the backbone network, and constructing the point cloud and BEV fusion features containing multi-scale and multi-level information.

[0021] As a further limitation of the first aspect of the application, in the fusion segmentation head, first, the multi-scale BEV feature is mapped to the point cloud space through the back projection operation, and is aligned with the corresponding point cloud feature through the multi-layer perception, and then the two are fused through the addition operation to obtain the fused feature of the joint representation; the fused feature is fused again with the initial point cloud feature in the encoding stage to enhance the detail expression ability of the point-level feature; the fused point cloud feature passes through a linear layer to directly generate semantic prediction results for each point, realizing three-dimensional semantic segmentation.

[0022] As a further limitation of the first aspect of the application, multiple auxiliary BEV segmentation heads are introduced, and cross-entropy loss, Lovász-Softmax loss and boundary loss are jointly used to construct a multiple supervision mechanism to guide the backbone network to learn more robust intermediate features, thereby improving the semantic prediction accuracy and boundary perception ability.

[0023] As a further limitation of the first aspect of the application, the loss calculation formula is as follows:

[0024] Let the prediction result output by the main segmentation head be F p , and the outputs of the four auxiliary segmentation heads be

[0025] The loss function of the main segmentation head and the loss function of the auxiliary segmentation head are defined as follows:

[0026]

[0027] Where λ1, λ2, and are the weight coefficients of the loss function; Q p is the ground truth of the point cloud, Q b is the ground truth of each BEV pixel, which is determined by the votes of all points projected into the pixel region; and represent the cross-entropy loss, Lovász-Softmax loss and boundary loss, respectively;

[0028] The final total loss function is the sum of the main loss and all auxiliary losses:

[0029]

[0030] Secondly, the application provides a three-dimensional point cloud segmentation system based on bidirectional fusion of point cloud and bird's eye view, comprising:

[0031] An acquisition module is configured to acquire original point cloud data to be segmented.

[0032] The segmentation module is configured to process the obtained original point cloud data by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network, and a fusion segmentation head; the encoder module is configured to extract an initial feature of each point and project it to a BEV space, generate a preliminary BEV feature after processing by a two-dimensional convolution layer, then project it back to the point cloud space, fuse it with the initial feature of the point cloud, and project the fused point cloud feature to the BEV space again, and further fuse it with the BEV feature obtained by the previous processing, thereby forming a preliminary fusion representation of the point cloud and the BEV; the backbone network takes the preliminary fusion representation of the point cloud and the BEV as input, adopts an alternating stacking structure of multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules, and realizes bidirectional interaction enhancement of the point cloud and BEV features layer by layer, and outputs point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is configured to integrate multi-level features and realize high-precision semantic prediction to obtain a final segmentation result.

[0033] In a third aspect, the present application provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implement the three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described in the first aspect.

[0034] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, the processor and the memory being in communication with each other, the memory storing program instructions executable by the processor, and the processor invoking the program instructions to execute the three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described in the first aspect.

[0035] In a fifth aspect, the present application provides an electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute instructions for implementing the three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described in the first aspect.

[0036] Terminology explanation:

[0037] Three-dimensional point cloud (3D Point Cloud): a collection of discrete points in space collected by sensors such as laser radars, each point containing three-dimensional coordinates and optional intensity, reflectivity, etc. attributes, used to represent the structural information of the physical environment.

[0038] Bird’s Eye View (BEV): A representation of projecting a 3D point cloud onto a 2D plane (usually the ground plane), preserving the spatial layout of the scene, suitable for efficient processing with 2D convolutional networks.

[0039] 3D Point Cloud Segmentation: The process of assigning semantic class labels to each point in a point cloud, used to understand and recognize different objects or regions in a 3D scene.

[0040] mIoU (Mean Intersection over Union): A commonly used performance evaluation metric in semantic segmentation tasks, measuring the overlap between predicted results and true labels. The higher the value, the better the segmentation effect.

[0041] Advantages of the present invention: The proposed bidirectional fusion mechanism of point cloud and bird’s eye view effectively improves the model’s ability to express local and global features through the interactive fusion of point cloud and BEV features at multiple stages, significantly improving the problems of fuzzy semantic boundaries and small target recognition difficulties, and significantly outperforming existing BEV methods in segmentation accuracy. At the same time, the computational graph is kept lightweight in structure design, which has significant advantages in parameter quantity and inference speed compared to voxel methods and multi-representation fusion methods, better balancing accuracy and real-time performance, suitable for scenarios such as autonomous driving that require high efficiency.

[0042] The advantages of the additional aspects of the present invention will be more apparent in the description below or through the practice of the present invention. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 Model structure diagram of the prior art method based on original point cloud.

[0045] Figure 2 Model structure diagram of the prior art method based on voxel.

[0046] Figure 3 Model structure diagram of the prior art method based on projection.

[0047] Figure 4 Model structure diagram of the prior art method based on multi-representation fusion.

[0048] Figure 5 A three-dimensional point cloud segmentation method model structure diagram based on bidirectional fusion of point clouds and bird's eye view according to an embodiment of the present application.

[0049] Figure 6 A structure diagram of a bidirectional fusion attention module according to an embodiment of the present application.

[0050] Figure 7 A subjective experiment comparison result diagram of a method PolarNet based on only bird's eye view projection in a SemanticKITTI data set according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] Embodiments of the present application are described below in detail with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar components are denoted by the same or similar reference numerals throughout. The embodiments described below with reference to the drawings are exemplary and are for the purpose of explanation only, and are not to be construed as limiting the present application.

[0052] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0053] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as such.

[0054] Those skilled in the art can understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the use of the phrase "comprising" in the specification of the present application means that the features, integers, steps, operations, elements and / or groups thereof are present, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0055] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Those skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0056] In order to make the present application more comprehensible, the present application will be further explained and described with specific examples in connection with the accompanying drawings, and the specific examples do not constitute limitations to the embodiments of the present application.

[0057] It should be understood by those skilled in the art that the drawings are only schematic views of the embodiments, and the components in the drawings are not necessarily essential for the implementation of the present application.

[0058] The present application proposes a three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view. The method enhances the feature expression ability of point cloud in its spatial neighborhood by designing a BEV-point cloud joint encoder; introduces a bidirectional fusion attention module to realize the deep interaction and fusion of point cloud features and BEV features at multiple levels; at the same time, a fusion segmentation head is constructed to integrate multi-scale features to improve the point-level prediction accuracy. This method effectively overcomes the defects of existing BEV methods in information loss and incomplete representation, balances segmentation accuracy and inference efficiency, and is suitable for real-time scenarios such as autonomous driving and intelligent transportation.

[0059] Embodiment 1

[0060] In this embodiment 1, first, a three-dimensional point cloud segmentation system based on bidirectional fusion of point cloud and bird's eye view is provided, including: an acquisition module for acquiring original point cloud data to be segmented. A segmentation module is used to process the acquired original point cloud data using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model includes an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them to the BEV space, generate preliminary BEV features after processing by a two-dimensional convolution layer, then project them back to the point cloud space, and fuse them with the initial features of the point cloud. The fused point cloud features are projected to the BEV space again and further fused with the BEV features obtained by previous processing, thereby forming the preliminary fusion representation of point cloud and BEV; the backbone network takes the preliminary fusion representation of point cloud and BEV as input, adopts the structure of alternating stacking of multi-layer two-dimensional residual convolution block and bidirectional fusion attention module, realizes the bidirectional interaction enhancement of point cloud and BEV features layer by layer, and outputs the point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

[0061] In this embodiment, the above system is used to realize a three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view, which comprises: obtaining original point cloud data to be segmented; processing the obtained original point cloud data by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them to the BEV space, generate preliminary BEV features after two-dimensional convolution layer processing, then project them back to the point cloud space, fuse the initial features of the point cloud, and further fuse the fused point cloud features to the BEV features obtained by the previous processing, thereby forming the preliminary fusion representation of point cloud and BEV; the backbone network takes the preliminary fusion representation of point cloud and BEV as input, adopts the structure of alternately stacking multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules, realizes the bidirectional interaction enhancement of point cloud and BEV features layer by layer, and outputs point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

[0062] In the encoder module, the calculation formula is as follows:

[0063]

[0064] wherein, is the original point cloud, MLP(·) represents a fully connected layer containing batch normalization and Relu activation function; Conv(·) represents a 3x3 convolution layer containing batch normalization and HSwish activation function; P2B(·) represents projecting point cloud features to BEV representation, wherein multiple points can be mapped to the same pixel position; B2P(·) represents projecting BEV features to point representation, wherein points falling into the same pixel position share the same features; Cat(·) represents tensor concatenation operation.

[0065] The backbone network is composed of multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules stacked alternately, aiming to realize multi-level fusion of point cloud and BEV features. The preliminary fusion representation of point cloud and BEV is first input into the backbone network. In each layer, the BEV features are first processed by the two-dimensional residual convolution block to extract the semantic information of the layer, and then input into the bidirectional fusion attention module together with the corresponding point cloud features to realize the interactive fusion of point cloud and BEV.

[0066] In the bidirectional fusion attention module, first, the BEV feature of the current layer is projected back to the point cloud space through spatial mapping, and is concatenated with the point cloud feature of the current layer in the channel dimension to form a cross-modal joint representation. The concatenated feature is aligned through a multi-layer perceptron (MLP) to output an enhanced point cloud feature to improve its modeling capability of global context information. Then, the enhanced point cloud feature is projected back to the BEV space, concatenated with the current layer BEV feature, and input into a two-dimensional convolution network for fusion processing to strengthen the local structure expression of the BEV feature.

[0067] Further, a multi-level feature attention module (HFA) is introduced to enhance the fused BEV feature in a fine-grained manner. The module is composed of a channel attention, a spatial attention and a semantic attention in series, which model the channel response intensity, the key spatial region and the cross-layer semantic consistency respectively. The three attentions are integrated through a weighted fusion mechanism to act on different dimensions of the BEV feature, achieving the strengthening of semantic key information and the suppression of redundant information. Finally, the enhanced BEV feature is added element by element to the output of the original convolution path in the backbone network to realize residual connection.

[0068] The point cloud feature and the BEV feature output by the current layer will be input into the next layer bidirectional fusion attention module and two-dimensional residual convolution block respectively to realize the layer-by-layer fusion and feature progression of the point cloud and the BEV in the backbone network, and to construct the point cloud and BEV fusion feature containing multi-scale and multi-level information.

[0069] In the fusion segmentation head, first, the multi-scale BEV feature is mapped to the point cloud space through the back projection operation, and is aligned with the corresponding point cloud feature through the multi-layer perceptron, and then the two are fused through addition operation to obtain the fusion feature of the joint representation; the fusion feature is fused again with the initial point cloud feature in the encoding stage to enhance the detail expression ability of the point-level feature; the fused point cloud feature is directly generated semantic prediction results for each point through a linear layer to realize three-dimensional semantic segmentation. Multiple auxiliary BEV segmentation heads are introduced, and cross-entropy loss, Lovász-Softmax loss and boundary loss are used jointly to construct a multiple supervision mechanism to guide the backbone network to learn more robust intermediate features, thereby improving the semantic prediction accuracy and boundary perception ability.

[0070] The loss calculation formula is as follows:

[0071] Let the prediction result output by the main segmentation head be F p and the outputs of the four auxiliary segmentation heads be

[0072] The loss functions of the main segmentation head and the auxiliary segmentation heads are defined as follows:

[0073]

[0074] where λ1, λ2, and are the weight coefficients of the loss function; Q p is the ground truth of the point cloud, Q b is the ground truth of each BEV pixel, which is determined by the votes of all points projected into the pixel area; and represent the cross-entropy loss, Lovász-Softmax loss and boundary loss, respectively;

[0075] The final total loss function is the sum of the main loss and all auxiliary losses:

[0076]

[0077] Embodiment 2

[0078] The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view proposed in this embodiment is implemented using a pre-trained segmentation model, and the segmentation model can complete training and testing on a single 2080Ti GPU.

[0079] The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view proposed in this embodiment designs a BEV-point cloud joint encoder to enhance the feature expression ability of points in their spatial neighborhood, solves the problem of local detail loss in BEV representation; constructs a bidirectional fusion attention module to realize deep interaction between point cloud features and BEV features, effectively alleviates the problems of projection distortion and semantic boundary blur; introduces a fusion segmentation head and a multi-level feature supervision mechanism to improve the segmentation accuracy and boundary perception ability of the model, while considering real-time inference efficiency, suitable for high-precision scene perception tasks such as autonomous driving.

[0080] The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view proposed in this embodiment has a specific model structure as shown in Figure 5 , which aims to solve the problem of segmentation accuracy decline caused by information loss in BEV representation, and enhance the perception ability of the model to local details and global structure, so as to improve the segmentation effect and consider real-time performance.

[0081] First, input the original point cloud into the BEV-point cloud joint encoder module designed by us, as shown in Figure 5As shown in the BEV-point cloud joint encoder module part, the encoder first extracts the initial features of each point using a multi-layer perception (MLP); then, the initially extracted point cloud features are converted into a BEV representation through a projection transformation, and after being processed by a two-dimensional convolution layer, preliminary BEV features are generated, which are then back-projected into the point cloud space and fused with the initial features of the point cloud to enhance the global context representation of the points; finally, the enhanced point cloud features are projected into the BEV space again, and the BEV features obtained through the previous processing are further fused to form a preliminary fusion representation of the point cloud and the BEV, which serves as the input for subsequent network processing.

[0082] The calculation formula of the first step is as follows:

[0083]

[0084] wherein is the original point cloud, MLP(·) represents a fully connected layer containing batch normalization and a Relu activation function; Conv(·) represents a 3x3 convolution layer containing batch normalization and a HSwish activation function; P2B(·) represents the projection of point cloud features into a BEV representation form, wherein multiple points may be mapped to the same pixel position, so we retain the most significant features through a max-pooling operation; B2P(·) represents the back-projection of BEV features into a point representation form, wherein points falling into the same pixel position share the same features; Cat(·) represents the tensor concatenation operation.

[0085] In the second step, the fused features are input into a backbone network composed of multiple two-dimensional residual convolution blocks for efficient extraction of BEV features. In each residual block, a bidirectional fusion attention module is introduced, which has a structure as shown in Figure 6 , realizing the layer-by-layer interactive fusion of BEV features and point cloud features. Among them, on the one hand, the BEV features at different scales are back-projected into the point cloud space to enhance the point cloud's perception of global context; on the other hand, the point cloud features are projected back into the BEV space to strengthen the local detail expression of the BEV features. In order to further suppress the noise interference caused by the sparsity of the point cloud in the BEV representation, a multi-level feature attention module (HFA) is designed to jointly model at the channel, spatial, and semantic levels, effectively improving the semantic consistency and segmentation discriminability of the BEV features. Finally, the attention-enhanced BEV features are residual fused with the original output of the backbone network, serving as the input for the subsequent segmentation head.

[0086] In the third step, a fusion segmentation head is designed to integrate multi-level features and achieve high-precision semantic prediction. As shown in Figure 5As shown in the fusion segmentation head part, different levels of features have different expression capabilities: low-level features have stronger spatial structure representation, and high-level features contain more rich semantic information. Therefore, the point cloud features and BEV features at different stages in the backbone network are fused to make full use of multi-scale context information. First, the multi-scale BEV features are mapped to the point cloud space through the back projection operation, and are aligned with the corresponding point cloud features through the multi-layer perception, and then the two are fused by addition operation to obtain the fused features of the joint representation. To further compensate for the possible loss of fine-grained structure information in the backbone network, the fused features are fused with the initial point cloud features at the encoding stage again to enhance the detail expression ability of the point-level features.

[0087] Finally, the fused point cloud features are directly generated semantic prediction results for each point through a linear layer, realizing end-to-end three-dimensional semantic segmentation without post-processing.

[0088] At the same time, multiple auxiliary BEV segmentation heads are introduced, and cross-entropy loss, Lovász-Softmax loss and boundary loss are used jointly to construct a multiple supervision mechanism to guide the backbone network to learn more robust intermediate features, thereby improving the semantic prediction accuracy and boundary perception ability. The loss calculation formula is as follows:

[0089] Let the prediction result output by the main segmentation head be F p And the outputs of the four auxiliary segmentation heads are respectively The loss function of the main segmentation head and the loss function of the auxiliary segmentation head are defined as follows:

[0090]

[0091] Where λ1, λ2, and are the weight coefficients of the loss function. Q p is the ground truth of the point cloud, Q b is the ground truth of each BEV pixel, which is determined by the votes of all points projected into the pixel region. and represent the cross-entropy loss, Lovász-Softmax loss and boundary loss respectively. The final total loss function is the sum of the main loss and all auxiliary losses:

[0092]

[0093] Figure 7Visualization results of two methods on the SematicKITTI dataset are shown, where the leftmost is the ground truth, and the right two columns show the segmentation results of PolarNet and our method, respectively. To emphasize the difference from the ground truth, correct segmentation and incorrect segmentation are marked in gray and red, respectively. Through subjective evaluation and objective comparison, the proposed scheme in this study significantly improves the segmentation effect of small moving targets such as cars, bicycles, motorcycles, and trucks, and the boundary division of large flat areas such as green belts, roads, and sidewalks is clearer. Table 1 shows the objective experimental results on the SemanticKITTI test set and the NuScenes validation set.

[0094] Table 1

[0095]

[0096] FPS represents the inference speed on a 2080Ti GPU.

[0097] Embodiment 3

[0098] This embodiment 3 provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implement a three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described above, which comprises:

[0099] Obtaining original point cloud data to be segmented;

[0100] Processing the obtained original point cloud data using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network, and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them to the BEV space, generate preliminary BEV features after two-dimensional convolution layer processing, and then project them back to the point cloud space, fuse the initial features of the point cloud, and the fused point cloud features are projected to the BEV space again, and further fused with the BEV features obtained by previous processing, thereby forming the preliminary fusion representation of point cloud and BEV; the backbone network takes the preliminary fusion representation of point cloud and BEV as input, adopts the structure of alternating stacking of multi-layer two-dimensional residual convolution block and bidirectional fusion attention module, and realizes the bidirectional interaction enhancement of point cloud and BEV features layer by layer, and outputs point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

[0101] Embodiment 4

[0102] The embodiment 4 provides a computer device, comprising a memory and a processor, the processor and the memory are in communication with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described above, which comprises:

[0103] obtaining original point cloud data to be segmented;

[0104] processing the obtained original point cloud data by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network and a fusion segmentation head; the encoder module is used for extracting initial features of each point and projecting them to a BEV space, generating preliminary BEV features after two-dimensional convolution layer processing, and then back-projecting them back to the point cloud space, fusing the initial features of the point cloud, and further fusing the fused point cloud features to the BEV space and the BEV features obtained by previous processing, so as to form a preliminary fusion representation of the point cloud and the BEV; the backbone network takes the preliminary fusion representation of the point cloud and the BEV as input, adopts a structure of alternately stacking multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules, realizes bidirectional interaction enhancement of point cloud and BEV features layer by layer, and outputs point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used for integrating multi-level features and realizing high-precision semantic prediction to obtain a final segmentation result.

[0105] Embodiment 5

[0106] The embodiment 5 provides an electronic device, comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the instructions for realizing the three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view as described above, which comprises:

[0107] obtaining original point cloud data to be segmented;

[0108] The obtained original point cloud data is processed by using a pre-trained segmentation model to obtain a segmentation result; wherein the segmentation model comprises an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial feature of each point and project it to the BEV space, generate the preliminary BEV feature after processing by the two-dimensional convolution layer, and then project it back to the point cloud space, fuse it with the initial feature of the point cloud, and then project the fused point cloud feature to the BEV space again, and further fuse it with the BEV feature obtained by the previous processing, so as to form the preliminary fusion representation of the point cloud and the BEV; the backbone network takes the preliminary fusion representation of the point cloud and the BEV as input, adopts the structure of alternately stacking multiple two-dimensional residual convolution blocks and bidirectional fusion attention modules, and realizes the bidirectional interaction enhancement of the point cloud and BEV features layer by layer, and outputs the point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

[0109] In summary, the present application first proposes a three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's eye view. In view of the problems of information loss and insufficient local features in BEV representation, a BEV-point cloud joint encoder is designed. Through feature projection and back projection operations, the feature interaction between point cloud and BEV is realized, and the global representation ability of points is enhanced. The present application also proposes a bidirectional fusion attention module to dynamically fuse point cloud and BEV features at each backbone network stage, effectively alleviating the noise interference caused by projection distortion, and improving the perception ability of the model to semantic boundaries and small objects. At the same time, a fusion segmentation head and a multi-loss supervision mechanism are designed to remove the cumbersome post-processing operation, improve the segmentation accuracy and boundary expression ability of the model, and achieve high-precision point-level semantic segmentation results while ensuring inference efficiency. The existing segmentation methods based on original point cloud, voxelization and multi-representation fusion generally have the problems of complex model, large computational overhead and difficulty in meeting real-time requirements. Although the method based on bird's eye view (BEV) representation performs well in computational efficiency, due to the information compression and structure distortion in the projection process of three-dimensional point cloud, the feature expression is incomplete, it is difficult to accurately perceive fine-grained structures, and the segmentation accuracy is affected. The present application proposes a bidirectional fusion mechanism of point cloud and bird's eye view to effectively improve the expression ability of the model to local and global features through the interactive fusion of point cloud and BEV features at multiple stages, significantly improve the problems of semantic boundary blur and small target recognition difficulty, and significantly outperform the existing BEV methods in segmentation accuracy. At the same time, the present application keeps the computational graph lightweight in structure design, and has significant advantages in parameter quantity and inference speed compared with the voxel method and the multi-representation fusion method, which can better balance the accuracy and real-time performance, and is suitable for scenarios such as autonomous driving that require high efficiency. In addition to the present application, existing methods such as PVD (Point-to-Voxel Knowledge Distillation) based on knowledge distillation can also improve the inference efficiency of the point cloud segmentation model without significantly reducing the performance. This method extracts the structure knowledge of point level and voxel level from the pre-trained teacher model, and introduces difficult example sampling and similarity constraint to guide the lightweight student model to learn the structure information of the environment, so as to balance the accuracy and efficiency.

[0110] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied therein.

[0111] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block or blocks in the block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0112] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flows and / or block or blocks in the block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0113] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block or blocks in the block diagrams. Figure 1 one or more flows and / or blocks. Figure 1 one or more blocks.

[0114] The above description is only a specific implementation of the present application, and is not intended to limit the protection scope of the present application. It should be understood by those skilled in the art that various modifications or changes can be made on the basis of the disclosed technical solutions without any inventive effort, and all these modifications or changes should be covered within the protection scope of the present application.

Claims

1. A 3D point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view, characterized in that: include: Obtain the original point cloud data to be segmented; The acquired raw point cloud data is processed using a pre-trained segmentation model to obtain a segmentation result; wherein, the segmentation model includes an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them into the BEV space, generate preliminary BEV features after processing by a two-dimensional convolution layer, and then back-project them back into the point cloud space and fuse them with the initial features of the point cloud. The fused point cloud features are projected back into the BEV space again and further fused with the previously processed BEV features to form a preliminary fusion representation of the point cloud and BEV; the backbone network takes the preliminary fusion representation of the point cloud and BEV as input, and adopts a structure in which multiple layers of two-dimensional residual convolution blocks and bidirectional fusion attention modules are alternately stacked to realize bidirectional interactive enhancement of the point cloud and BEV features layer by layer, and output point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

2. The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view according to claim 1 is characterized in that: The calculation formula in the encoder module is as follows: in, For the original point cloud, MLP(·) represents a fully connected layer with batch normalization and Relu activation function; Conv(·) represents a 3×3 convolutional layer with batch normalization and HSwish activation function; P2B(·) represents the projection of point cloud features to BEV representation, where multiple points may be mapped to the same pixel position; B2P(·) represents the back projection of BEV features to point representation, where points falling into the same pixel position share the same features; Cat(·) represents tensor weighting operation.

3. The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view according to claim 1 is characterized in that: The backbone network consists of multiple layers of alternating two-dimensional residual convolution blocks and bidirectional fusion attention modules, aiming to achieve multi-level fusion of point cloud and BEV features. The initial fusion of point cloud and BEV means that they are first input into the backbone network. In each layer, the BEV features are first subjected to the two-dimensional residual convolution block to extract the semantic information of the layer, and then are input into the bidirectional fusion attention module together with the corresponding point cloud features to achieve interactive fusion of point cloud and BEV. In the bidirectional fusion attention module, the BEV features of the current layer are first back-projected to the point cloud space through spatial mapping and spliced ​​with the point cloud features of the current layer in the channel dimension to form a cross-modal joint representation. The spliced ​​features are aligned through a multi-layer perceptron, and the enhanced point cloud features are output to improve its ability to model global context information. Subsequently, the enhanced point cloud features are projected back to the BEV space, spliced ​​with the BEV features of the current layer, and input into a two-dimensional convolutional network for fusion processing to strengthen the local structural expression of the BEV features. A multi-level feature attention module is introduced to perform fine-grained enhancement on the fused BEV features. This module consists of three branches: channel attention, spatial attention, and semantic attention, which are connected in series to model channel response strength, key spatial regions, and cross-layer semantic consistency, respectively. The point cloud features and BEV features output by this layer will be used as the input of the bidirectional fusion attention module and the two-dimensional residual convolution block of the next layer respectively, realizing the layer-by-layer fusion and feature progression of the point cloud and BEV in the backbone network, and constructing a point cloud-BEV fusion feature containing multi-scale and multi-level information.

4. The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view according to claim 1 is characterized in that: In the fusion segmentation head, first, the multi-scale BEV features are mapped to the point cloud space through a back-projection operation, and aligned with the corresponding point cloud features through a multi-layer perceptron. Then, the two are fused by an addition operation to obtain a fusion feature of joint representation; the fusion feature is fused again with the initial point cloud features in the encoding stage to enhance the detail expression ability of the point-level features; the fused point cloud features are passed through a linear layer to directly generate semantic prediction results for each point to achieve three-dimensional semantic segmentation.

5. The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view according to claim 1 is characterized in that: Multiple auxiliary BEV segmentation heads are introduced, and cross entropy loss, Lovász-Softmax loss and boundary loss are used jointly to build a multi-supervision mechanism to guide the backbone network to learn more robust intermediate features, thereby improving semantic prediction accuracy and boundary perception ability.

6. The three-dimensional point cloud segmentation method based on bidirectional fusion of point cloud and bird's-eye view according to claim 1 is characterized in that: The loss calculation formula is as follows: The prediction result output by the main segmentation head is recorded as F p , and the outputs of the four auxiliary segmentation heads are recorded as The loss function of the main segmentation head and the loss function of the auxiliary segmentation head are defined as follows: Among them, λ1, λ2, and is the weight coefficient of the loss function; Q p is the true value of the point cloud, Q b is the true value of each BEV pixel, which is determined by the votes of all points projected into the pixel area; and Represent cross entropy loss, Lovász-Softmax loss and boundary loss respectively; The final total loss function is the sum of the main loss and all auxiliary losses:

7. A 3D point cloud segmentation system based on bidirectional fusion of point cloud and bird's-eye view, characterized in that: include: An acquisition module is used to obtain the original point cloud data to be segmented; The segmentation module is used to process the acquired raw point cloud data using a pre-trained segmentation model to obtain a segmentation result; wherein, the segmentation model includes an encoder module, a backbone network and a fusion segmentation head; the encoder module is used to extract the initial features of each point and project them into the BEV space, generate preliminary BEV features after processing by a two-dimensional convolution layer, and then back-project them back into the point cloud space and fuse them with the initial features of the point cloud. The fused point cloud features are projected back into the BEV space again and further fused with the previously processed BEV features to form a preliminary fusion representation of the point cloud and BEV; the backbone network takes the preliminary fusion representation of the point cloud and BEV as input, and adopts a structure in which multiple layers of two-dimensional residual convolution blocks and bidirectional fusion attention modules are alternately stacked to realize bidirectional interactive enhancement of the point cloud and BEV features layer by layer, and output point cloud and BEV fusion features containing multi-scale and multi-level information; the fusion segmentation head is used to integrate multi-level features and realize high-precision semantic prediction to obtain the final segmentation result.

8. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the three-dimensional point cloud segmentation method based on the bidirectional fusion of point cloud and bird's-eye view according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the three-dimensional point cloud segmentation method based on the bidirectional fusion of point cloud and bird's-eye view as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the three-dimensional point cloud segmentation method based on the bidirectional fusion of point cloud and bird's-eye view as described in any one of claims 1 to 6.

Images