A semantic segmentation method for road point clouds based on orthogonal pyramid dual attention network

Abstract

This application relates to the field of point cloud processing technology and discloses a road point cloud semantic segmentation method based on an orthogonal pyramid dual-attention network. This method introduces an enhanced dual-attention orthogonal branch module into the encoder network, utilizing dynamic orthogonalization to decouple and fuse local geometric features and global semantic features. Then, through a cross-layer multi-scale information interaction module, features from each layer of the encoder are aggregated into multi-scale fused features as a global context prior. In the decoder network, a pyramid-gated fusion module is used to intelligently fuse the contextual features of the decoding path with the detailed features of the encoder under the guidance of the global context prior. This invention solves the problems of redundant feature representation, insufficient cross-layer information interaction, and low accuracy in decoding detail recovery in existing technologies, improving the accuracy and robustness of semantic segmentation in complex road scenes, and showing significant effects in distinguishing similar objects and reconstructing object boundaries.

Description

Technical Field

[0001] This invention relates to the field of point cloud processing technology, specifically to a road point cloud semantic segmentation method based on an orthogonal pyramid dual attention network. Background Technology

[0002] Point clouds, as the core data carrier of the three-dimensional world, are fundamental to key applications such as autonomous driving environmental perception, high-precision map construction, and robot navigation through point-by-point semantic segmentation. In recent years, the encoder-decoder architecture based on deep learning has become the mainstream technical route for point cloud semantic segmentation. This architecture extracts the abstract features of the point cloud layer by layer through the encoder network, and then gradually recovers the point-by-point prediction results through the decoder network.

[0003] However, existing technologies still have inherent limitations when dealing with complex scenes. During feature encoding, the extraction of local geometric structure information and the perception of global semantic context information are often intertwined, leading to redundancy or even conflicts in the generated feature representations. This reduces the model's accuracy in distinguishing objects with similar local shapes but different semantic categories. Furthermore, in the information transfer between the encoder and decoder, conventional skip connection mechanisms typically only occur between corresponding layers. The highly abstract semantic information captured by the deep encoder is difficult to effectively guide the feature recovery process of the shallow decoder, creating a semantic gap and affecting the global consistency of the segmentation results.

[0004] Finally, in the decoding stage, when fusing the upsampled contextual features with the detailed features from the encoder, simple splicing or addition operations are often used. This non-adaptive fusion method cannot intelligently filter and retain effective information that is crucial for boundary recovery, often leading to problems such as blurred object edges and loss of fine structures, which limits the precision of the final segmentation result.

[0005] Therefore, this invention proposes a road point cloud semantic segmentation method based on an orthogonal pyramid dual attention network to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a road point cloud semantic segmentation method based on an orthogonal pyramid dual attention network. This method solves the problems that road point cloud semantic segmentation methods often face when dealing with complex scenes, such as entanglement between local geometric details and global semantic context features, insufficient cross-level feature information interaction, and low detail recovery accuracy during decoding.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a road point cloud semantic segmentation method based on an orthogonal pyramid dual attention network, comprising the following steps:

[0008] The input point cloud data is hierarchically feature-encoded by an encoder network, which includes an enhanced dual attention orthogonal branch module to decouple and fuse local geometric features and global semantic features, and output multi-scale features at different levels.

[0009] By aggregating multi-scale features from different levels through a cross-layer multi-scale information interaction module, multi-scale fused features are formed.

[0010] The encoder features are hierarchically decoded by a decoder network. Guided by multi-scale fusion features, the decoder network uses a pyramid gating fusion module to fuse the context features of the decoding path with the detailed features of the encoder network to generate the final features.

[0011] The final features are mapped to point-by-point semantic categories to complete the semantic segmentation of the road point cloud.

[0012] Preferably, the enhanced dual attention orthogonal branch module includes:

[0013] The local geometry module is used to extract local geometric features from the input features;

[0014] A semantically guided sparse global attention module is used to extract global semantic features from input features;

[0015] An adaptive orthogonal weighted fusion module is used to perform dynamic orthogonalization and weighted fusion on the local geometric features and global semantic features.

[0016] Preferably, the adaptive orthogonal weighted fusion module performs dynamic orthogonalization in the following way:

[0017] Based on the local geometric features and global semantic features, an adaptive orthogonal control scalar is generated;

[0018] Calculate the projection components of the local geometric features onto the global semantic features;

[0019] The orthogonalized local features are obtained by subtracting the projection components weighted by the adaptive orthogonal control scalar from the local geometric features.

[0020] Preferably, the semantically guided sparse global attention module is specifically used for:

[0021] Online semantic clustering is performed using the local geometric features to divide the point cloud into several feature clusters;

[0022] A sparse attention mask is generated based on the clustering results. The mask value between point pairs that are classified into the same feature cluster is a value that makes the corresponding attention weight approach zero after Softmax normalization. The mask value between point pairs that are classified into different feature clusters is a preset negative value.

[0023] Multi-head attention computation is performed under the guidance of the sparse attention mask to generate global semantic features.

[0024] Preferably, the cross-layer multi-scale information interaction module is specifically used for:

[0025] Collect the multi-scale features output by all encoder layers in the encoder network, and unify the multi-scale features to a preset spatial resolution and channel dimension to obtain a set of aligned feature maps.

[0026] Based on the aligned feature maps, generate point-by-point fusion weights for each feature map;

[0027] The aligned feature maps are weighted and summed using the point-by-point fusion weights to obtain multi-scale fusion features.

[0028] Preferably, the pyramid gating fusion module includes:

[0029] The gated skip connection fusion module is used to perform gated fusion of upsampled features from the decoding path and skip connection features from the encoder network to obtain preliminary fused features.

[0030] The dynamic local pyramid fusion module is used to perform multi-scale local feature extraction and dynamic weighted fusion on the preliminary fused features to obtain refined features.

[0031] The preliminary fusion features and the refined features are added together through residual connection to obtain the output of the pyramid gating fusion module.

[0032] Preferably, the gated jump connection fusion module is specifically used for:

[0033] Based on the upsampling features from the decoding path and the skip connection features from the encoder network, point-by-point gate values ​​are generated;

[0034] The upsampled features and skip connection features are weighted and summed using the gating value.

[0035] Preferably, the dynamic local pyramid fusion module is specifically used for:

[0036] The preliminary fused features are processed by a set of parallel local feature extractors with different receptive fields to generate a set containing multiple local scale features.

[0037] Based on multiple local scale features, point-by-point fusion weights are generated for each local scale feature.

[0038] The refined features are obtained by weighting and summing multiple local scale features using the point-by-point fusion weights.

[0039] Preferably, at least one of the parallel local feature extractors is a deformable alignment module, which predicts the three-dimensional spatial offset of the neighborhood sampling points based on the preliminary fusion features, and performs feature aggregation based on the new position after adjustment of the three-dimensional spatial offset.

[0040] The method further includes a preprocessing step for the input point cloud data, the preprocessing step including:

[0041] Construct an initial high-dimensional feature vector for each point in the point cloud;

[0042] Generate a neighborhood index for each point in the point cloud;

[0043] Generate a hierarchical downsampling index for each layer of the encoder network.

[0044] This invention provides a road point cloud semantic segmentation method based on an orthogonal pyramid dual-attention network. It has the following beneficial effects:

[0045] 1. This invention employs an enhanced dual attention orthogonal branch module to perform dynamic orthogonalization processing on local geometric features and global semantic features during the feature encoding stage. This effectively removes common information between the two types of features, enabling the network to learn more discriminative decoupled feature representations. This improves the model's ability to distinguish objects with similar local structures but different semantic categories, such as accurately distinguishing roadside utility poles and tree trunks in road scenes.

[0046] 2. This invention designs a cross-layer multi-scale information interaction module to align and dynamically weight the feature maps of all layers in the encoder network, forming a unified multi-scale fusion feature. This feature is provided as a high-level global context prior information to each layer of the decoding network, effectively solving the semantic gap problem between the encoding and decoding paths, enhancing the model's ability to understand large-scale objects and complex scene structures, thereby improving the overall consistency of the segmentation results.

[0047] 3. The present invention employs a pyramid-gated fusion module in the decoder, replacing the traditional simple feature stitching method. This module intelligently filters detailed features from the encoder through a gating mechanism and uses a dynamic local pyramid structure to refine the fused features at multiple scales, achieving deep adaptive fusion of contextual and detailed information. This enables the network to more accurately reconstruct the boundary contours of objects and retain fine geometric details while restoring the feature map resolution. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the system implementation environment of the present invention;

[0049] Figure 2 This is a flowchart illustrating the point cloud semantic segmentation method of the present invention.

[0050] Among them, 100 is a computing device; 110 is a processor module; 120 is a memory module; 130 is a storage module; 140 is a data interface module; and 200 is a data acquisition device. Detailed Implementation

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] See attached document Figure 1 , Figure 1 This is a schematic diagram of the system implementation environment according to an embodiment of the present invention; the embodiment of the present invention provides a road point cloud semantic segmentation system based on an orthogonal pyramid dual attention network, which can be used to execute the point cloud semantic segmentation method described later; it can be specifically implemented as a computing device;

[0053] The computing device 100 includes a processor module 110, a memory module 120, a storage module 130, and a data interface module 140; these modules communicate via an internal bus or other forms of connection mechanism.

[0054] The processor module 110, as the core computing unit of the computing device 100, is used to execute computer program instructions stored in the storage module 130 to implement the point cloud semantic segmentation method of the present invention. In a specific embodiment, the processor module 110 may include one or more central processing units (CPUs) and one or more graphics processing units (GPUs). The graphics processing unit (GPU) is suitable for performing large-scale matrix and tensor operations in deep learning networks due to its inherent parallel computing capabilities.

[0055] The memory module 120, such as random access memory (RAM), provides runtime data cache space for the processor module 110; during the execution of the point cloud semantic segmentation method, intermediate calculation results, point cloud data batches, and weight parameters of the network model can be loaded here;

[0056] Storage module 130, such as solid-state drive (SSD) or hard disk drive (HDD), is used for persistent storage of data; its specific storage contents include operating system, deep learning framework (such as PyTorch or TensorFlow), and computer program code containing the implementation of the method of the present invention; in addition, the original point cloud dataset and the trained network model file are also stored in this module;

[0057] The data interface module 140 is used to realize data communication between the computing device 100 and external devices; for example, the interface may be an Ethernet interface, a Controller Area Network (CAN) interface, or a Universal Serial Bus (USB) interface.

[0058] In one embodiment of the present invention, the data interface module 140 is connected to a data acquisition device 200; the data acquisition device 200, such as one or more LiDAR sensors, is responsible for scanning the physical environment and generating raw point cloud data containing information such as three-dimensional coordinates and reflection intensity; the generated data is input to the computing device 100 for processing through the data interface module 140.

[0059] See attached document Figure 2 , Figure 2 This is a flowchart illustrating a point cloud semantic segmentation method according to an embodiment of the present invention; the present invention provides a point cloud semantic segmentation method applied to a computing device 100, which specifically includes the following steps:

[0060] Step S201: Acquire and preprocess input point cloud data; This step receives raw point cloud data generated by data acquisition device 200 and performs structured processing on it to generate initial feature vectors, neighborhood indexes, and hierarchical downsampling indexes for subsequent network processing.

[0061] Step S202: Perform hierarchical feature encoding on the preprocessed point cloud data; in this step, an encoder network containing multiple encoder layers is used, and an enhanced dual attention orthogonal branch module is used to extract and abstract multi-scale features from the point cloud data, while decoupling and fusing local geometric information and global semantic information at each scale.

[0062] Step S203: Perform cross-layer multi-scale information interaction; In this step, a cross-layer multi-scale information interaction module is used to aggregate the multi-scale features output from different layers in the encoder network to form a unified feature representation rich in contextual information, and provide it to the subsequent decoding steps.

[0063] Step S204: Perform hierarchical feature decoding on the encoded features; in this step, the spatial resolution of the point cloud is gradually restored through a decoder network containing multiple decoder layers and a pyramid gated fusion module, while intelligently fusing detailed features from the encoder network and contextual features from the decoding path itself.

[0064] Step S205: Generate the final semantic prediction result; In this step, the final feature map output by the last decoder layer is input into a classification head to calculate the semantic category of each point in the original point cloud, thereby completing the semantic segmentation task of the entire point cloud scene.

[0065] To enable those skilled in the art to better understand the technical solution of the present invention, the specific implementation of each of the above steps will be described in detail below with reference to the accompanying drawings.

[0066] In the embodiments of the present invention, the specific implementation of step S201, acquiring and preprocessing the input point cloud data, is described in detail; the purpose of this step is to convert the unstructured raw point cloud data into a structured data format suitable for deep neural network processing.

[0067] Step S201-1: Receive raw point cloud data; the computing device 100 acquires a point cloud scene from the data acquisition device 200 (e.g., LiDAR) through its data interface module 140; the acquired raw point cloud data is represented as a set of points, where each point contains at least three-dimensional spatial coordinates (X, Y, Z); in some embodiments, each point may also contain additional physical attribute information, such as the reflection intensity of the laser echo;

[0068] Step S201-2: Construct initial feature vectors; since the original 3D coordinates are not numerically suitable for direct use as input features of deep neural networks, they need to be mapped to a more expressive high-dimensional feature space; for the first feature vector in the point cloud... First, determine the three-dimensional coordinates of each point. With optional additional attribute information (For example, reflection intensity) is concatenated along the feature dimension to form an initial low-dimensional feature vector; subsequently, this low-dimensional feature vector is fed into a feature mapping network. Processing is performed to generate initial high-dimensional feature vectors. Feature mapping network The specific implementation is a multilayer perceptron (MLP); this process can be represented by the following formula:

[0069] ;

[0070] in, Representing the The initial high-dimensional feature vector of each point Represents a feature mapping network. Representing the The three-dimensional coordinate vector of a point Representing the Additional attribute vectors for each point This represents a concatenation operation along the feature dimension.

[0071] Step S201-3: Generate key index information; In order to support the efficient calculation of local feature aggregation and hierarchical downsampling in subsequent network layers, two key index information needs to be pre-calculated and generated: neighborhood index and hierarchical downsampling index.

[0072] The neighborhood index is generated to define the local neighborhood range of each point. In a specific embodiment of the present invention, the K-Nearest-Neighbors (KNN) algorithm is used to determine the neighborhood. For each point in the point cloud, the algorithm searches for the point with the closest spatial distance in the entire point cloud set according to a preset distance metric. Record these points. The index of each point; the distance metric can be three-dimensional Euclidean distance; the generated neighborhood index provides the computational range for the subsequent local feature extraction module;

[0073] The generation of hierarchical downsampling indexes is used to construct a multi-level pyramid structure for the encoder network. In one specific embodiment of the invention, the Farthest-Point-Sampling (FPS) algorithm is used to generate the downsampling point set. For each layer of the encoder, the FPS algorithm iteratively selects sampling points from the input point set of that layer. In each iteration, the point furthest from all selected sampling points is added to the sampling point set until the preset number of sampling points for that layer is reached. This process generates a set of indices for each layer of the encoder, indicating the mapping relationship from the point set of the previous layer to the point set of the current layer. In another embodiment, other sampling algorithms such as Random-Sampling (RS) can also be used to replace the Farthest-Point-Sampling algorithm.

[0074] Step S201-4: Encapsulate a structured data object; organize and encapsulate all the data generated in the previous steps to form a unified, structured data object as input to the subsequent neural network model; this data object specifically includes: a set of three-dimensional coordinates of all points in the original point cloud, a set of initial high-dimensional feature vectors corresponding to all points, a set of neighborhood indices corresponding to all points, and a set of hierarchical downsampling indices corresponding to each level of the encoder; this data object is implemented as a set of tensors, the format of which is compatible with the deep learning framework used, thereby realizing the efficient transfer of data to the network model.

[0075] In this embodiment of the invention, the specific implementation of step S202, which performs hierarchical feature encoding on the preprocessed point cloud data, is described in detail. The core of this step lies in the application of an Enhanced-Dual-Attention-Orthogonal-Branch (E-DAOB) module. The encoder network consists of multiple stacked encoder layers, each of which contains an E-DAOB module and a downsampling operation for reducing spatial resolution.

[0076] Existing technologies typically employ simple concatenation or element-wise addition when fusing local geometric features and global semantic features. Such operations implicitly assume that the two features are complementary, but fail to address the common correlations and information redundancy between the two features, thus limiting the expressive power of the final fused features. To address this issue, the E-DAOB module proposed in this invention adopts a design concept of parallel processing and explicit decoupling.

[0077] The E-DAOB module consists of three cooperative sub-modules: a local geometry module, a semantically guided sparse global attention module, and an adaptive orthogonal weighted fusion module. When processing input features, the local geometry module and the semantically guided sparse global attention module form two parallel processing branches, which are used to capture the fine local geometric structure of the point cloud and the long-range global semantic dependencies, respectively. Subsequently, the adaptive orthogonal weighted fusion module is responsible for intelligently fusing the features output by these two parallel branches.

[0078] Specifically, the core innovation of the E-DAOB module lies in its feature fusion mechanism, which achieves intelligent decoupling and complementary fusion of local and global features through dynamic orthogonalization. Feature decoupling refers to actively and explicitly eliminating the parts of local features that overlap with or can be explained by global features before fusion.

[0079] This process is achieved through a dynamic orthogonalization operation: the local feature vector is projected onto the direction of the global feature vector, and an adjustable part of the projected component is subtracted from the original local feature vector, thereby obtaining an orthogonalized local feature that is complementary to the global feature.

[0080] The intelligence or adaptability is reflected in the fact that the strength of orthogonalization is not fixed, but dynamically determined by a data-driven control module based on the characteristics of the current input features. In this way, the E-DAOB module ensures that the final fused features are highly complementary and information-efficient, avoiding the negative impact of feature redundancy and improving the quality of feature representation. The implementation details of each sub-module of this module will be described below:

[0081] The local geometry module in the E-DAOB module aims to process the features input to the current encoder layer to capture the refined local geometric structure contained in the point cloud data. The data processing of this local geometry module, for the input features... any point in Specifically, this may include the following steps:

[0082] Step S202-1: Relative relationships in the encoding space; for points and its neighborhood determined in step S201-3 any neighboring point in The difference in their three-dimensional coordinates is calculated to obtain a relative position vector; this relative position vector is then input into an encoding network. A nonlinear transformation is performed to generate a high-dimensional relative position embedding. This process aims to map the geometric spatial relationships between point pairs to a feature space that facilitates network understanding.

[0083] ;

[0084] in, Representative point and Embedding of relative positions between them and These represent the three-dimensional coordinate vectors of two points. The representative coding network, in one specific embodiment, is a multilayer perceptron (MLP).

[0085] Step S202-2: Generate spatial attention weights; relative position embedding Used to calculate neighborhood points For the center point The importance of attention, i.e., attention weight; specifically, the importance of attention weight. Input into an attention score calculation network This yields a scalar attention score. Subsequently, regarding the center point The attention scores of all neighboring points are normalized using the Softmax function to obtain the final attention weights. ;

[0086] ;

[0087] ;

[0088] in, The unnormalized attention score represents the score. The attention score calculation network, in one specific embodiment, is a multilayer perceptron; Represents the normalized attention weights. Representative point The set of neighborhood indexes; The index representing the center point currently being processed in the point cloud; Representative center point The index of a certain neighboring point; This represents the standard exponential function in mathematics. Representative center point Another point in its neighborhood The unnormalized attention scores between them; This represents a traversal index used to traverse the neighborhood index set. All elements in the (i.e., the indices of all neighboring points).

[0089] Step S202-3: Aggregate neighborhood features and output; utilize the attention weights generated in the previous step. Input features of neighboring points We perform a weighted summation to obtain features that aggregate neighborhood geometric information. Subsequently, the aggregation feature is compared with the center point. Its own input features The components are combined through a residual connection and then passed through a nonlinear activation function. The processing ultimately generates the local geometric features of that point. ;

[0090] ;

[0091] ;

[0092] in, and Representing points respectively and Input features, Represents the neighborhood features after aggregation. The local geometric features representing the final output This represents a non-linear activation function, such as the rectified linear unit (ReLU). It means that for all satisfying The conditions are accumulated.

[0093] Repeat the above steps for all points in the input point set to obtain the complete local geometric feature map. ; This local geometric feature map It serves a dual purpose: firstly, as the output of a parallel branch of the E-DAOB module, it provides information rich in fine geometric structure for the final feature fusion; secondly, it will be used as prior knowledge input for the subsequent semantically guided sparse global attention module; due to local geometric features... The point cloud has already undergone preliminary structuring and semantic grouping through a local attention mechanism (e.g., points belonging to the same physical surface will have more similar features). Therefore, it can provide effective guidance information for the global attention mechanism to determine which long-distance dependencies between point pairs are worth computing, thereby achieving sparsity and efficiency in global attention computation.

[0094] In the E-DAOB module, parallel to the local geometry module is a semantically guided sparse global attention module. This sparse global attention module aims to efficiently capture long-range dependencies in point cloud scenes, i.e., global context information, while overcoming the quadratic computational complexity problem brought about by standard global attention mechanisms. Its specific implementation includes the following steps:

[0095] Step S202-4: Generate query, key, and value matrices; for attention calculation, the input features need to be... This is converted into three different representations: query, key, and value. First, the input features are optionally processed through a linear layer. Channel compression is performed to reduce computational overhead; subsequently, a learnable positional code is applied. The compressed features are added element-wise to inject absolute or relative spatial location information of points into the features, resulting in location-aware features. Finally, through three independent linear mapping layers , , ,Will Projecting them separately, we obtain the query matrix. Key matrix Sum matrix ;

[0096] ;

[0097] ;

[0098] in, This represents the input features of the encoder layer. Represents position code, This represents a feature that has been injected with location information; , , They represent the components used for generation. , , The weight matrix of the linear mapping layer;

[0099] Step S202-5: Generate a sparse mask using a sparse attention scheduler; To avoid ineffective attention calculations for all point pairs in the scene, this invention introduces a sparse attention scheduler, whose core function is to generate a sparse attention mask based on semantic prior information. The scheduler utilizes the local geometric features generated in step S202-3. As semantic guidance information; due to It already contains the local structural information of the points, and points with similar features are semantically or structurally related;

[0100] Specifically, the scheduler internally applies a clustering algorithm to analyze local geometric features. Perform fast online semantic clustering; in one specific embodiment, the clustering algorithm can be a lightweight K-means clustering algorithm; the clustering process divides the point cloud set into several feature clusters, and the points in each cluster are considered semantically related because of their similar local geometric features;

[0101] Based on the clustering results, a sparse attention mask is generated. ; It is a matrix with the same size as the attention score matrix to be calculated; the assignment rule for its elements is: if the first element is the attention score matrix to be calculated... The point and the first If points are assigned to the same feature cluster, then the elements at the corresponding positions in the matrix... The value is set to 0; conversely, if the two points do not belong to the same cluster, the element at that position is set to a negative number with a large absolute value (e.g., negative infinity). In this way, the mask directs computational resources to key point pairs within the cluster. In another embodiment, to allow limited cross-cluster information interaction, the mask generation rules can be configured to also allow attention computation between representative points (e.g., cluster centroids) of different clusters.

[0102] Step S202-6: Perform sparse multi-head attention computation; this step involves the sparse attention mask. Guided by this, the global context features are calculated; to capture feature dependencies in different subspaces, a multi-head attention mechanism is adopted; firstly, , , The matrix is ​​uniformly divided along the feature dimension into Each part corresponds to One attention point, denoted as .

[0103] For each attention head, the attention score is calculated as follows: Calculate and The dot product of the transpose of the expression is then scaled; subsequently, the sparse attention mask generated in the previous step is applied... Add element-wise to the scaled dot product result; because The values ​​at non-clustered point pairs are large negative numbers. After processing with the Softmax function, the attention weights at these positions will approach zero, thus achieving sparse attention computation. Finally, the obtained attention weights are compared with... Multiply them to get the output of the head;

[0104] ;

[0105] in, Representing the The output of each attention head, It is the dimension of the key vector. As a scaling factor, it is used to maintain gradient stability. Represents the matrix transpose operation. It is a normalized exponential function; Representing the A query matrix for each attention head; Representing the A key matrix for each attention head; Representing the A matrix of values ​​for each attention head.

[0106] Step S202-7: Aggregate multi-head outputs; combine all... The output of each attention head The data is concatenated along the feature dimension and then passed through a final linear mapping layer. By fusing the features, a unified global semantic feature is obtained. ;

[0107] ;

[0108] in, Represents the final generated global semantic features. Representing multiple attention points, the first The output feature of the last attention head. This represents a concatenation operation along the feature dimension. This represents the weight matrix of the output linear mapping layer; this global semantic feature This is the final output of the parallel branch, which captures the semantically relevant long-distance dependencies in the scene.

[0109] After obtaining local geometric features (From steps S202-3) and global semantic features (From step S202-7) After that, the E-DAOB module fuses the two features through an adaptive orthogonal weighted fusion module to generate the final output features of the encoder layer. This process aims to explicitly eliminate redundant information between two features, achieving complementary fusion; its specific implementation includes the following steps:

[0110] Step S202-8: Generate an adaptive orthogonal control scalar; to achieve dynamic control of the feature orthogonalization intensity, a pointwise adaptive orthogonal control scalar must first be generated. Specifically, local geometric features With global semantic features The features are concatenated along the feature dimension, and the concatenated features are then input into an orthogonal control network. Orthogonal control network The output is processed by a Sigmoid activation function. This generates a control scalar with a value between 0 and 1 for each point in the point cloud. ;

[0111] ;

[0112] in, Represents local geometric features, Represents global semantic features. This represents a concatenation operation along the feature dimension. This represents an orthogonal control network, which in one specific embodiment is a multilayer perceptron; Represents the Sigmoid activation function; controls the scalar. The value of determines the strength of subsequent orthogonalization operations.

[0113] Step S202-9: Perform dynamic feature orthogonalization; the purpose of this step is to transform local geometric features. In the middle, remove its connection with global semantic features Overlapping information components; first, calculate local geometric features. In global semantic features Projected components This projection component represents It can be The part represented by the line;

[0114] ;

[0115] in, represent and The dot product of the transpose of , represent The square of the L2 norm; then, using the adaptive orthogonal control scalar generated in the previous step. The orthogonalization process is dynamically adjusted; specifically, it is based on the original local geometric features. Subtract from Weighted projection components This yields an orthogonalized local feature. ;

[0116] ;

[0117] Through this operation Retained Zhongyu Orthogonal, complementary information; when When the value is 1, perform complete orthogonalization; when... When the value is 0, orthogonalization is not performed, and the original value is preserved. ;when When the values ​​are between 0 and 1, partial orthogonalization is performed; this data-driven dynamic adjustment mechanism enables the network to autonomously determine the degree of redundancy elimination based on the context of specific features.

[0118] Step S202-10: Perform dynamic weighted fusion; obtain orthogonalized local features that are complementary to the global features. Then, it is combined with global semantic features through a dynamic weighting mechanism. To perform the final fusion; first, to and The features are concatenated along the feature dimension, and the concatenated features are then input into a fusion weight generation network. The output of this network is processed by a Softmax function, thus providing... and Generate point-by-point fusion weights respectively and ;

[0119] ;

[0120] in, The fusion weight generation network, in one specific embodiment, is a 1x1 convolutional layer; and The weights are generated point-by-point, and satisfy the following condition at each point: Finally, the generated dynamic weights are used to perform a weighted sum of the two features to obtain the final output features of the E-DAOB module. ;

[0121] ;

[0122] This output feature Because it integrates locally and globally features that have been explicitly decoupled and have complementary information, it has higher expression efficiency and discrimination ability.

[0123] In this embodiment of the invention, the specific implementation of step S203, which involves cross-layer multi-scale information interaction, is described in detail. This step is implemented through a cross-layer multi-scale information interaction module (Skip-Connection-Pyramid-Block, SCP), which aims to effectively aggregate features extracted by the encoder at different abstraction levels, providing comprehensive prior information rich in context and detail for the subsequent decoding process.

[0124] Step S203-1: Collect and integrate multi-scale features; the encoder network includes Each encoder layer, after undergoing the hierarchical feature encoding step S202, will generate a sequence of encoder layers. A feature pyramid composed of feature maps of different scales is denoted as . ;in, Representing the The encoder layers output feature maps; these feature maps have different spatial resolutions (i.e., the number of points) and feature dimensions; in order to perform fusion, all feature maps need to be unified to the same spatial resolution.

[0125] In one specific embodiment, the spatial resolution of the encoder's first layer (i.e., the layer with the highest resolution) is selected as the target resolution; for Feature map Upsampling is used to restore its resolution to the same level as... Consistency; the upsampling operation can be implemented based on the downsampling index generated in step S201-3, through methods such as nearest neighbor interpolation or three-point linear interpolation; in another embodiment, the channel dimensions of each feature map can be adjusted by a 1x1 convolution after upsampling to keep them consistent.

[0126] Step S203-2: Perform adaptive pyramid fusion; merging all feature maps at all scales By unifying the spatial resolution and channel dimensions, a set of aligned feature maps is obtained. Then, through an adaptive pyramid fusion mechanism, they are fused into a unified feature representation.

[0127] First, all aligned feature maps By concatenating along the feature dimensions, a wide-channel feature map that aggregates information from all scales is formed. Subsequently, the aggregated feature map is input into a fusion weight generation network. To generate point-by-point fusion weights corresponding to features at each scale;

[0128] ;

[0129] ;

[0130] in, This represents a concatenation operation along the feature dimension; The representative weight generation network, in one specific embodiment, is a network with a number of output channels. A 1x1 convolutional layer; It is a weight tensor that contains a weight at each point. A weight vector of dimensions, generated by the Softmax function, ensuring that the sum of the weights at each scale is 1; The standard Softmax function;

[0131] Step S203-3: Generate and transfer multi-scale fusion features; utilize the dynamic weights generated in the previous step. The weighted summation of all aligned feature maps yields the final multi-scale fused feature. ;

[0132] ;

[0133] in, Represents the weight tensor The middle corresponds to the first The weighted components are calculated on an element-wise basis; the weighted summation operation is performed on an element-wise basis. Represents the encoder's first The layer outputs a feature map that has been upsampled and optionally adjusted for channel dimensions to align it with feature maps at all other scales in terms of spatial resolution and number of channels.

[0134] Multi-scale fusion features It is rich in information ranging from low-level details to high-level semantics, and through a data-driven adaptive weighting method, the network can autonomously determine which scale of information should be emphasized at different spatial locations; As a high-level prior information, it will be passed to each subsequent decoder layer, providing rich contextual guidance for the hierarchical feature decoding steps, thereby helping the decoder to maintain semantic consistency while recovering details.

[0135] In this embodiment of the invention, the specific implementation of step S204, which performs hierarchical feature decoding of the encoded features, is described in detail. This step is implemented through a decoder network consisting of multiple stacked decoder layers, wherein each decoder layer contains an upsampling operation and a Pyramid-Gated-Fusion-Block (PGFB) module. PGFB aims to intelligently balance and fuse semantic information from deep networks (decoding paths) with geometric detail information from shallow networks (encoder skip connections) to enhance the expressive power of features while gradually restoring the spatial resolution of the point cloud.

[0136] For any decoder layer in the decoder network, its PGFB module receives two input features: one is the upsampled feature from a deeper decoder layer. This feature is rich in high-level semantic contextual information; the other path is the skip connection feature from the corresponding layer in the encoder network with the same spatial resolution. This feature preserves richer local geometric details; the PGFB module's processing of these two features may include the following steps:

[0137] Step S204-1: Perform gated jump connection fusion; for adaptive balancing and The contribution is made by employing a data-driven gating mechanism; firstly, the two input features are concatenated along the feature dimension, and then the concatenated features are input into a gating network. The network's output is activated by a Sigmoid function. Generate a gating value between 0 and 1 for each point in the point cloud. ;

[0138] ;

[0139] in, This represents a gated network, which in one specific embodiment is a 1x1 convolutional layer; This represents the concatenation operation along the feature dimension; subsequently, this gate value is used. The two features are weighted and fused to obtain a preliminary fused feature. ;

[0140] ;

[0141] This gating mechanism allows the network to autonomously decide, based on the specific context of each point, whether to rely more on semantic information from the decoding path (when...). When the value is close to 0, it still relies more on detailed information from the encoder regarding skipped connections (when the value is close to 0). (Approaching 1).

[0142] Step S204-2: Construct and merge dynamic local pyramids; to further enhance the initial fusion features. The ability to express details and perceive context is refined by constructing a local feature pyramid; firstly, a set of parallel local feature extractors with different receptive fields are used to extract details and context. Processing is performed to capture its features at multiple local scales; in one specific embodiment, this set of extractors is for different neighborhood sizes. (For example The K-Nearest Neighbor (KNN) feature aggregation module generates a set containing multiple local scale features. ;

[0143] To improve adaptability to irregular geometric shapes of point clouds, in another embodiment, the local feature extractor can be implemented as a deformable alignment module. Before aggregating neighborhood features for the center point, the deformable alignment module uses a small network to predict the three-dimensional spatial offset of the neighborhood sampling points based on the features of the center point, and performs feature interpolation and aggregation based on these new offset positions. This deformable mechanism allows the feature extraction range to better fit the actual shape of the target.

[0144] After obtaining a multi-scale set of local features, a dynamic weighting mechanism is used to fuse them; all local scale features are then fused. The features are concatenated along the feature dimension, and the concatenated features are then input into a pyramid weight generation network. The network's output is processed by a Softmax function to generate point-by-point fusion weights for each local scale. ;

[0145] ;

[0146] in, The pyramid weight generation network, in one specific embodiment, is a 1x1 convolutional layer.

[0147] Step S204-3: Generate the final output features; utilize the generated dynamic weights By performing a weighted summation of all local scale features, a feature refined through multiple scales is obtained. Finally, this refined feature is compared with the module's input. The final output features of the PGFB module are obtained by summing the results through a residual connection. ;

[0148] ;

[0149] ;

[0150] in, The multi-scale refined features are obtained by dynamically weighting and fusing multiple local scale features; Represents the range from 1 to Summing all local scales, It is the total number of scales of the constructed local feature pyramid; Represents dynamic weights, which is the first... Point-by-point weight values ​​generated from local scale features for weighted summation; Representing the The local scale feature is derived from the first... Features generated by a local feature extractor with a specific receptive field; This represents the final output feature of the pyramid-gated fusion module, which is also the output feature of the decoder layer it resides in; Representing the initial fusion features, these are intermediate features generated by the gated skip connection fusion step, which fuse upsampled features and skip connection features.

[0151] This output feature It retains deep semantics while effectively absorbing and enhancing shallow details through gating mechanisms and dynamic pyramid fusion, thus providing high-quality feature representations for the final semantic prediction.

[0152] In this embodiment of the invention, the specific implementation of step S205, which generates the final semantic prediction result, is described in detail. This step receives the final feature representation from the last layer of the decoder network and converts it into a point-by-point semantic category prediction.

[0153] Step S205-1: Map features to categories using the classification head; convert the final feature map output by the decoder network. The input is fed into a classification head; the classification head is a multilayer perceptron (MLP), whose function is to map high-dimensional point feature vectors to a vector representing the scores of each semantic category.

[0154] In one specific embodiment, the classification head consists of one or more linear layers, normalization layers, and nonlinear activation function layers stacked alternately, and finally connected to a final output linear layer; the number of output channels of the final output linear layer is set to the total number of semantic categories to be segmented. Therefore, for the input point cloud The output of this classification head is a point with a dimension of 1. The matrix, namely the logits matrix. Elements in the matrix Represents the first The point belongs to the th point The original predicted scores for each category are used; when performing inference to obtain the final segmentation result, the maximum index (argmax) operation is applied to each row (i.e. each point) of the matrix to obtain the final semantic category label for that point.

[0155] Step S205-2: Calculate the loss function during the network training phase; to optimize the parameters of the entire neural network, a loss function needs to be defined and calculated during training to quantify the difference between the network prediction results and the true labels; in one embodiment of the present invention, weighted cross-entropy loss is used as the loss function; this loss function can effectively handle the common class imbalance problem in point cloud data, that is, the number of points of some classes (such as roads) is much greater than that of other classes (such as pedestrians).

[0156] Specifically, for a containing The input point cloud of points has a weighted cross-entropy loss. The calculation method is as follows:

[0157] ;

[0158] in: It is the total number of points in the point cloud; It is the total number of semantic categories; It is the index of the point; It is the point output from the classification head in step S205-1. Category Logical fractions; It is a point The true semantic category label; It is a specific logical score, referring to the model's response to the first... Each point belongs to its true category. The predicted score; Represents an exponential function; Represents the natural logarithm function; It is a preset, and category Associated weights; these weights are used to adjust the contribution of different categories in the total loss calculation; expression It is the result of the Softmax function calculation, which converts the logistic score into a probability value, representing the model's prediction of the first... Each point belongs to its true category. The probability of.

[0159] Category weight The frequency of each category in the training dataset can be pre-calculated. One specific implementation is to use a weighting scheme that is inversely proportional to the category frequency to increase the influence of rare categories in the loss calculation.

[0160] In another embodiment, the loss function can also be implemented in other forms, such as Focal-Loss, which reduces the weight of well-classified samples by modifying the standard cross-entropy loss, so that the model can focus more on samples that are difficult to classify; or, Lovasz-Softmax loss can be used, which directly optimizes the Intersection-over-Union (IoU) ratio, a commonly used evaluation metric in segmentation tasks.

[0161] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A road point cloud semantic segmentation method based on orthogonal pyramid dual attention network, characterized in that, Includes the following steps: The input point cloud data is hierarchically feature-encoded by an encoder network, which includes an enhanced dual attention orthogonal branch module to decouple and fuse local geometric features and global semantic features, and output multi-scale features at different levels. By aggregating multi-scale features from different levels through a cross-layer multi-scale information interaction module, multi-scale fused features are formed. The encoder features are hierarchically decoded by a decoder network. Guided by multi-scale fusion features, the decoder network uses a pyramid gating fusion module to fuse the upsampled features of the decoding path with the detailed features of the encoder network to generate the final features. The final features are mapped to point-by-point semantic categories to complete the semantic segmentation of the road point cloud.

2. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 1, characterized in that, The enhanced dual attention orthogonal branch module includes: The local geometry module is used to extract local geometric features from the input features; A semantically guided sparse global attention module is used to extract global semantic features from input features; An adaptive orthogonal weighted fusion module is used to perform dynamic orthogonalization and weighted fusion on the local geometric features and global semantic features.

3. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 2, characterized in that, The adaptive orthogonal weighted fusion module performs dynamic orthogonalization in the following way: Based on the local geometric features and global semantic features, an adaptive orthogonal control scalar is generated; Calculate the projection components of the local geometric features onto the global semantic features; The orthogonalized local features are obtained by subtracting the projection components weighted by the adaptive orthogonal control scalar from the local geometric features.

4. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 2, characterized in that, The semantically guided sparse global attention module is specifically used for: Online semantic clustering is performed using the local geometric features to divide the point cloud into several feature clusters; A sparse attention mask is generated based on the clustering results. For point pairs that are divided into different feature clusters, the element value at the corresponding position in the mask is negative infinity, so that the attention weight of the point pair approaches zero after Softmax normalization. For point pairs that are classified into the same feature cluster, the element value at the corresponding position in the mask is 0; Multi-head attention computation is performed under the guidance of sparse attention masks to generate global semantic features.

5. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 1, characterized in that, The cross-layer multi-scale information interaction module is specifically used for: Collect the multi-scale features output by all encoder layers in the encoder network, and unify the multi-scale features to a preset spatial resolution and channel dimension to obtain a set of aligned feature maps. Based on the aligned feature maps, generate point-by-point fusion weights for each feature map; The aligned feature maps are weighted and summed using the point-by-point fusion weights to obtain multi-scale fusion features.

6. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 1, characterized in that, The pyramid gating fusion module includes: The gated skip connection fusion module is used to perform gated fusion of upsampled features from the decoding path and skip connection features from the encoder network to obtain preliminary fused features. The dynamic local pyramid fusion module is used to perform multi-scale local feature extraction and dynamic weighted fusion on the preliminary fused features to obtain refined features. The preliminary fusion features and the refined features are added together through residual connection to obtain the output of the pyramid gating fusion module.

7. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 6, characterized in that, The gated jump connection fusion module is specifically used for: Based on the upsampling features from the decoding path and the skip connection features from the encoder network, point-by-point gate values ​​are generated; The upsampled features and skip connection features are weighted and summed using the gating value.

8. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 6, characterized in that, The dynamic local pyramid fusion module is specifically used for: The preliminary fused features are processed by a set of parallel local feature extractors with different receptive fields to generate a set containing multiple local scale features. Based on multiple local scale features, point-by-point fusion weights are generated for each local scale feature. The refined features are obtained by weighting and summing multiple local scale features using the point-by-point fusion weights.

9. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 8, characterized in that, At least one of the parallel local feature extractors is a deformable alignment module, which predicts the three-dimensional spatial offset of the neighborhood sampling points based on the preliminary fusion features, and performs feature aggregation based on the new position after adjustment of the three-dimensional spatial offset.

10. The road point cloud semantic segmentation method based on orthogonal pyramid dual attention network according to claim 1, characterized in that, The method further includes a preprocessing step for the input point cloud data, the preprocessing step including: Construct an initial high-dimensional feature vector for each point in the point cloud; Generate a neighborhood index for each point in the point cloud; Generate a hierarchical downsampling index for each layer of the encoder network.

Images