Point cloud segmentation method and device based on multi-scale feature extraction and fusion, and medium

Abstract

The application discloses a point cloud segmentation method and device based on multi-scale feature extraction and fusion and a medium. The method comprises the following steps: acquiring point cloud data, capturing features by using four kinds of scale convolution kernels of point level, small scale, medium scale and large scale according to different scale features, obtaining four feature map outputs of different scales, and fusing the four feature map outputs to obtain output features; performing adaptive fusion on different features extracted in the channel dimension by an SE module, expanding a receptive field, performing three-dimensional space feature fusion, and obtaining a feature map after adaptive feature selection; and performing once weighted addition on the original feature map and the feature map after adaptive feature selection, taking a weighted addition result as a final output according to a preset weighting coefficient. By improving a traditional semantic segmentation method and a deep learning method, the application can quickly and accurately complete an object point cloud segmentation problem in a multi-target complex scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of three-dimensional point cloud, and relates to a solution for semantic segmentation processing of three-dimensional point cloud, in particular to a point cloud segmentation method, device and medium based on multi-scale feature extraction and fusion. BACKGROUND

[0002] With the iteration of three-dimensional sensors and the development of three-dimensional data processing algorithms, it has become fast and efficient to obtain high-quality point cloud data indoors and outdoors. Point cloud data plays an important role in various real-world applications, from autonomous driving to augmented reality (AR). By processing point cloud data, the environment information can be better perceived. One of the key tasks of processing point cloud data is semantic segmentation of point cloud, which can promote the interaction of autonomous vehicles or AR applications with the real physical world. Research on three-dimensional point cloud semantic segmentation is of great significance to the research directions of smart city, high-precision map establishment, autonomous driving, and robot.

[0003] Current point cloud semantic segmentation methods can be mainly divided into four categories: point-based methods, voxel-based methods, projection-based methods, and hybrid representation-based methods. Point-based methods can be mostly regarded as extensions of PointNet and PointNet++. In order to reduce the computational cost when processing large-scale outdoor point clouds, the farthest sampling method is usually used to sample some key points, and then a series of variant point convolution operations are performed on the points within a given neighborhood to extract global and local context information according to the PointNet architecture. However, this method has two main drawbacks. First, the performance of these works is limited to some extent by the farthest sampling procedure; second, these methods mostly rely on k-nearest neighbor search to maintain local relationships between points in each frame, which increases the time complexity. One of the biggest advantages of voxel-based methods is that they can maintain the physical properties of point clouds and apply standard convolution layers. However, the voxelized point cloud itself is sparse, and non-empty grids only occupy a small part. Using dense convolution to process the discretized point cloud data is inefficient. Projection-based methods mainly include depth images and front view representations based on spherical projection. This regular, structured representation of data can directly apply a series of standard convolution layers and two-dimensional segmentation backbones to achieve a balance between efficiency and accuracy. However, the performance of these methods is closely related to the projection resolution and complex post-processing stages, which increases the additional computational cost. In order to improve the quality of point cloud segmentation, point cloud segmentation methods that learn multi-modal features have emerged. Hybrid representation-based methods emphasize the importance of representation integration and fusion, and methods that utilize both voxel feature learning and point feature learning improve segmentation performance at very high running efficiency. SUMMARY

[0004] The present application is provided to solve the above problems existing in the prior art. Therefore, a point cloud segmentation method, device and medium based on multi-scale feature extraction and fusion are needed, which applies multi-scale sparse convolution feature extraction and channel attention-based feature selection, and constructs a point cloud segmentation network framework based on two modules. The semantic segmentation method introduces multi-scale feature extraction sparse convolution, applies different size convolution kernels to capture point cloud feature information, and has the advantages of high computational efficiency, small computational amount, high interpretability, and high segmentation precision.

[0005] According to the first aspect of the present application, a point cloud segmentation method based on multi-scale feature extraction and fusion is provided, comprising:

[0006] According to different scale features, the obtained point cloud data is captured by point-level, small-scale, medium-scale and large-scale convolution kernels, respectively, to obtain four different scale feature map outputs x0, x1, x2 and x3;

[0007] The small-scale, medium-scale and large-scale feature map outputs are spliced together and sent to a Softmax activation function to obtain a spliced feature map:

[0008] U cat =cat(U1,U2,U3) (2)

[0009] In the formula, U cat ∈R N×3 , R represents a domain, N represents a point number, and 3 is a feature dimension, wherein the attention weight represented by the obtained different scales is:

[0010] S=softmax(U cat ) (3)

[0011] The features of different scales are element-wise added according to S∈R N×3 , thereby obtaining fused multi-scale features; the fused multi-scale features and the features captured by the point-level convolution kernel are element-wise added to obtain a new spliced feature, and the new spliced feature is subjected to a convolution operation to obtain an output feature O after self-selection and fusion of the features extracted according to the weights:

[0012] O=Conv{x0+x1×S[:,0]+x2×S[:,1]+x3×S[:,2]} (4)

[0013] Based on the output feature O, the SE module is used to adaptively fuse the different features extracted in the channel dimension, expand the receptive field, and perform three-dimensional spatial feature fusion to obtain a feature map after adaptive feature selection;

[0014] The original feature map and the feature map after adaptive feature selection are added by weighting, and the weighting addition result is taken as the final output according to a preset weighting coefficient.

[0015] Further, the small-scale convolution kernel is set to 1*1*1, the medium-scale convolution kernel is set to 2*2*2, the large-scale convolution kernel is set to 4*4*4, and the point-level convolution kernel is set to a sub-manifold convolution, the convolution kernel size of the sub-manifold convolution is 1, and the convolution step is set to 1.

[0016] Further, the method further comprises:

[0017] The small, medium and large scale feature map outputs x1, x2 and x3 are transmitted to a subsequent feature fusion module as the output of the multi-scale feature extraction module. In order to avoid using the same weight for different scale features, the large, medium and small scale features are added to obtain an intermediate feature map X, and then three different multi-layer perceptrons are used to transform the intermediate feature map X.

[0018] f i :X→U∈R N×4 (i=1,2,3) (1)

[0019] Each f i is a combination of different Convolutional layers (convolutional layers), Activation Function layers (activation function layers) and BatchNormalization layers (batch normalization layers).

[0020] Further, based on the output feature O, different features extracted are adaptively fused in the channel dimension by an SE module, the receptive field is expanded, three-dimensional spatial feature fusion is performed, and a feature map after adaptive feature selection is obtained, comprising:

[0021] The output feature O is subjected to global average pooling (Global Average Pooling), then the output 1*1*C data is subjected to two-level full connection operation, finally limited to the range of [0, 1] by using a Sigmoid activation function, and then the value is multiplied by C channels of O as an input feature of the next stage;

[0022] By controlling the size of the scale, important features are enhanced, adaptive selection of features contributing to the final result is performed, and unimportant features are weakened, thereby enhancing the directivity of the extracted features.

[0023] Further, the output feature O is globally averaged pooled, and then the output 1x1xC data is further operated by two-level full connection, and finally the calculation process of limiting to the range of [0, 1] by using a Sigmoid activation function is as follows:

[0024]

[0025] In the formula, δ represents a ReLU activation function, sigma represents a SigMoid activation function, u c (i,j) represents a feature at any point of an input feature map, H and L represent the height and width of the feature map, W represents a linear transformation, represents two linear transformations, and the number of feature channels is compressed from C to to obtain a result of reducing the model complexity, wherein r is a dimension reduction ratio coefficient; in order to restore the number of compressed channels in the first step to the original number of channels, a second linear transformation needs to be performed again, and finally the scale is controlled, and the attention weights obtained according to formula (3) are adaptively fused for different channel features.

[0026] Further, the weighting coefficient is set to 0.4.

[0027] According to the second technical solution of the present application, a point cloud segmentation device based on multi-scale feature extraction and fusion is provided, characterized in that the device comprises: an acquisition module configured to capture features by using four kinds of scale convolution kernels of point level, small scale, medium scale and large scale according to different scale features of acquired point cloud data, to obtain four different scale feature map outputs x0, x1, x2 and x3.

[0028] A fusion module is configured to splice the small scale, medium scale and large scale feature map outputs together, and send them into a Softmax activation function to obtain a spliced feature map as follows:

[0029] U cat =cat(U1,U2,U3) (2)

[0030] In the formula, U cat ∈R N×3 , R represents a domain, N represents a point number, and 3 is a feature dimension, wherein the attention weights represented by different scales are represented as follows:

[0031] S=softmax(U cat ) (3)

[0032] Different scale features are fused according to S∈R N×3Element-wise addition is performed to obtain the fused multi-scale features; element-wise addition is performed on the fused multi-scale features and the features captured by the point-level convolution kernel to obtain a new splicing feature, and the new splicing feature is subjected to a convolution operation to obtain the output feature O after different scale feature extraction and according to the weight self-selection fusion:

[0033] O = Conv{x0+x1xS[:,0]+x2xS[:,1]+x3xS[:,2]} (4)

[0034] The adaptive fusion module is configured to perform adaptive fusion on the extracted different features in the channel dimension based on the output feature O through the SE module, expand the receptive field, perform three-dimensional spatial feature fusion, and obtain the feature map after adaptive feature selection;

[0035] The calculation module is configured to perform a weighted addition on the original feature map and the feature map after adaptive feature selection, and according to the preset weighting coefficient, the weighted addition result is taken as the final output.

[0036] Further, the acquisition module is further configured to:

[0037] The small, medium and large scale feature maps x1, x2 and x3 are output as the output of the multi-scale feature extraction module and are delivered to the subsequent feature fusion module. In order to avoid using the same weight for different scale features, the large, medium and small scale features are added to obtain an intermediate feature map X, and then three different multi-layer perceptrons are used to transform the intermediate feature map X:

[0038] f i :X→U∈R N×4 (i = 1, 2, 3) (1)

[0039] Each f i is a combination of different Convolutional layers (convolutional layers), Activation Function layers (activation function layers) and BatchNormalization layers (batch normalization layers).

[0040] Further, the adaptive fusion module is further configured:

[0041] Global Average Pooling is performed on the output feature O, and then two-level fully connected operations are performed on the output 1x1xC data. Finally, the Sigmoid activation function is used to limit the range to [0, 1], and then the value is multiplied by the C channels of O as the input feature of the next level;

[0042] By controlling the size of the scale, important features are enhanced, features that contribute to the final result are adaptively selected, and unimportant features are weakened, thereby enhancing the directionality of the extracted features.

[0043] According to a third aspect of the present application, a non-transitory computer readable storage medium storing instructions which, when executed by a processor, perform the method according to various embodiments of the present application is provided.

[0044] The point cloud segmentation method, device and medium based on multi-scale feature extraction and fusion according to various aspects of the present application at least have the following technical effects:

[0045] 1. The object point cloud segmentation problem in a multi-target complex scene can be quickly and accurately completed by improving the traditional semantic segmentation and deep learning method.

[0046] 2. The application of multi-scale sparse convolution feature extraction and channel attention-based feature selection reduces the data amount of subsequent processing, enhances the features of interest of the point cloud, and improves the efficiency and segmentation accuracy of the network.

[0047] 3. The application reduces the entanglement between channel relationships and spatial relationships captured by convolution kernels in the process of performing convolution operations, enhances the difference between different feature vectors and the robustness of the segmentation algorithm, and improves the operation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0048] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. Like numerals having different letter suffixes can represent different instances of the like components. The drawings illustrate generally by way of example, and not by way of limitation, various embodiments of the application as contemplated herein. The same reference numerals are used to denote like components throughout the several views. Such embodiments do not, unless the context clearly dictates otherwise, represent an exhaustive list of all combinations or permutations of the described components.

[0049] Figure 1 An input point cloud multi-scale feature extraction flowchart in a point cloud segmentation method based on multi-scale feature extraction and fusion according to an embodiment of the present application is shown.

[0050] Figure 2 A point cloud feature fusion flowchart for different scales in a point cloud segmentation method based on multi-scale feature extraction and fusion according to an embodiment of the present application is shown.

[0051] Figure 3 An outdoor input point cloud visualization schematic diagram according to an embodiment of the present application is shown.

[0052] Figure 4 A flow chart of a point cloud segmentation method based on multi-scale feature extraction and fusion according to an embodiment of the present application is shown.

[0053] Figure 5 An outdoor point cloud ground segmentation schematic diagram according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be described in detail below in combination with the drawings and specific embodiments. The embodiments of the present application will be further described in detail below in combination with the drawings and specific embodiments, but not as a limitation on the present application. The order in which each step is described herein as an example should not be considered as a limitation, and those skilled in the art should know that the order can be adjusted, as long as the logic between them is not destroyed and the whole process cannot be realized.

[0055] The present application provides a point cloud segmentation method based on multi-scale feature extraction and fusion, as shown in Figure 1 and Figure 2 The method comprises the following steps:

[0056] Step one: according to different scale features, the point cloud data read in is captured by using four kinds of scale convolution kernels of point level, small scale, medium scale and large scale, wherein the small scale convolution kernel is set to (1*1*1), the medium scale convolution kernel is set to (2*2*2), the large scale convolution kernel is set to (4*4*4), and the point level convolution is set to sub-manifold convolution, that is, the convolution kernel size is 1, and the convolution step is set to 1. The four different scale convolution kernels are output from small to large, and are denoted as x0, x1, x2 and x3 respectively.

[0057] The small, medium and large scale feature maps x1, x2 and x3 are output as the output of the multi-scale feature extraction module and are transmitted to the subsequent feature fusion module. In order to avoid using the same weight for different scale features, the features of large scale, medium scale and small scale are added to obtain an intermediate feature map X, and then three different MLPs are used to transform the intermediate feature map X.

[0058] f i :X→U∈R N×4 (i=1,2,3) (1)

[0059] Each f iThe combination of different Convolutional layer, ActivationFunction layer and BatchNormalization layer.

[0060] Step two: The output features of different scales obtained in step one are spliced together and sent to a Softmax activation function, so that the spliced feature map is:

[0061] U cat = cat(U1, U2, U3) (2)

[0062] U cat ∈R N×3 (N represents the number of points), wherein the attention weight represented by the obtained different scales is:

[0063] S = softmax(U cat ) (3)

[0064] Then the features of different scales are element-wise added according to S ∈ R N×3 , so as to obtain the fused multi-scale features. The fused multi-scale features are element-wise added with the features obtained by the point cloud in step one and subjected to the sub-manifold sparse convolution, a new spliced feature is obtained, the addition result is subjected to a convolution operation again, and finally the self-selected fusion output O of the features extracted according to the weight is obtained.

[0065] O = Conv{x0+x1×S[:,0]+x2×S[:,1]+x3×S[:,2]} (4)

[0066] In formula (4), x1x2x3x0, respectively represent the outputs of different size convolution kernels and the output of the point-level sub-manifold sparse convolution.

[0067] Step three: In order to avoid the loss of high-dimensional information caused by multiple multi-layer convolution operations, the SE module proposed by Squeeze-and-Excitation Networks (SENet) is applied in step three to adaptively fuse the different features extracted in the channel dimension, expand the receptive field, and perform three-dimensional spatial feature fusion. In order to extend the SE module to 3D sparse convolution, the output feature O in step two is first subjected to a Global Average Pooling (i.e., the Squeeze process), and then the 1x1xC data output is subjected to two-level fully connected operations (i.e., the Excitation process). Finally, the Sigmoid activation function (i.e., the self-gating mechanism) is used to limit the value to the range of [0, 1], and then the value is multiplied as scale to the C channels of O as the input feature of the next level. By controlling the size of scale, important features are enhanced, and features that contribute to the final result are adaptively selected, and unimportant features are weakened, thereby enhancing the directionality of the extracted features. The Squeeze-Excitation calculation is as follows:

[0068]

[0069] In formula (5), δ represents the ReLU activation function, and σ represents the SigMoid activation function. c (i,j) represents the feature of the input feature map at any point, H and L represent the height and width of the feature map, and W represents the linear transformation, represents two linear transformations, and through the first linear transformation, the number of feature channels is compressed from C to to reduce the complexity of the model, where r is the dimension reduction ratio coefficient; in order to restore the number of compressed channels in the first step to the original number of channels, a second linear transformation is needed, and finally the scale is controlled. According to the attention weight calculated by formula (3), the features of different channels are adaptively fused.

[0070] Step four: After the adaptive selection and fusion of the features and weights in different channels in step three, the output of the expanded SE module is obtained. In order to retain the initial channel features in the final output result, the original feature map and the feature map after adaptive feature selection are subjected to a weighted addition, and the weighted addition result is set as the final output.

[0071] To verify the feasibility of the method described in this embodiment, the Semantic Kitti dataset and indoor point cloud data were selected as input data. The outdoor point cloud data range is x-axis: 105.988m, y-axis: 110.284m, z-axis: 30.528m. The dataset includes 22 sequences, totaling 43,551 frames of LiDAR data. Sequences 00 to 10 are the training set, providing the annotation file for each frame, while sequences 11 to 21 are the test set. In the experiments of this chapter, sequences 01 to 09 and 00 are used as the training set to train the model, and sequence 10 is used as the validation set to verify the segmentation effect of the model. Figure 3 A point cloud scene visualization is provided as the input test set. Based on the above data, the method is validated through the following steps:

[0072] Step 1: The read-in point cloud data is voxelized using a hash table-based downsampling module based on the voxel point cloud segmentation model. The number of point cloud samples in each batch is set to 3, the maximum number of iterations is 170,000, the initial learning rate is 0.01, the optimizer is SGD with momentum, and ExponentialLR is used as the learning rate adjustment strategy. The data augmentation methods used include 1) Flip; 2) Rotation; 3) Translations; 4) Random jitter; 5) Adding noise (Gaussian noise, etc.); 6) Radiometric transformation. It should be noted that the above data augmentation methods are only examples, and other methods can be used to achieve data augmentation. This embodiment does not impose specific limitations on them.

[0073] The point cloud data was processed using a hash table-based neighborhood retrieval and construction algorithm to build a hierarchical network structure with a time complexity of O(1). The proposed multi-scale feature extraction and fusion were then incorporated into the network, and the voxelization resolution was set to 0.04m. Figure 1 As shown, the input training set point cloud data is initially extracted using three voxels of different scales (1*1*1), (2*2*2), and (4*4*4) and a (1*1*1) submanifold convolution kernel to obtain features x0, x1, x2, and x3. x1, x2, and x3 are summed once, and then the same convolution operation is performed on the three features of different scales. The results are then concatenated to complete the first step of the initial feature extraction process.

[0074] Step two: multiply the point cloud data after the initial feature extraction with the first step feature extraction, optimize the extraction result, then add the elements of the submanifold convolution feature extraction to realize feature enhancement, and then use a convolution operation of (2*2*2) to extract compressed features. In order to improve the resolution of the predicted label, the voxel-level point cloud prediction category is converted to the point-level point cloud prediction category, and the nearest neighbor interpolation strategy is used to represent the core class label of each voxel as the class label of all discrete points in the voxel.

[0075] Step three: adaptive fusion is performed on the extracted point cloud feature data, as shown in the feature fusion flowchart. Figure 2 As shown in the feature fusion flowchart, the three scale features after convolution are multiplied by weight coefficients respectively, splicing operation is performed, and Squeeze-Excitation is used to strengthen the features of interest to enhance the features. The test classification point cloud graph after filtering the segmented ground is as shown in Figure 4 .

[0076] Step four: test the network segmentation effect after feature enhancement, as shown in Figure 5 , different colors of segmented objects represent different classes with different semantic information, and it can be seen that the segmentation accuracy and semantic information representation ability are improved.

[0077] Step five: test the indoor point cloud scene, change the initial learning rate parameter of the network setting to 0.1, use Warmup to preheat the learning rate, use the preheated Adagrad learning rate adjustment strategy in the first 500 iterations, and limit the input voxel number of the indoor scene to 50000. Get the test scene graph.

[0078] Step six: quantitative analysis of the semantic segmentation accuracy of the proposed method.

[0079] The initial point cloud data is segmented according to the semantics of cars, trucks, bicycles, trees, poles and other ground categories using the proposed method, and the mIOU is calculated by the following formula:

[0080]

[0081] In the formula, mIOU represents the average intersection over union, K represents the total number of categories, FN represents the number of positive samples predicted as negative samples, TP represents the number of positive samples predicted as positive samples, and FP represents the number of negative samples predicted as positive samples.

[0082] Thus, the semantic segmentation results of the proposed method are qualitatively analyzed. The semantic segmentation accuracy is shown in Table 1.

[0083]

[0084] Table 1

[0085] The embodiment of the present application also provides a point cloud segmentation device based on multi-scale feature extraction and fusion, which comprises:

[0086] An acquisition module is configured to capture features by using point-level, small-scale, medium-scale and large-scale convolution kernels respectively according to different scale features of acquired point cloud data, so as to obtain four feature map outputs of different scales, namely x0, x1, x2 and x3.

[0087] A fusion module is configured to splice the feature map outputs of the small-scale, medium-scale and large-scale together and send them into a Softmax activation function, so as to obtain a spliced feature map as follows:

[0088] U cat =cat(U1,U2,U3) (2)

[0089] In the formula, U cat ∈R N×3 , R represents a domain, N represents a point number, 3 is a feature dimension, wherein the attention weight represented by the different scales obtained is represented as:

[0090] S=softmax(U cat ) (3)

[0091] Different scale features are elementally added according to S∈R N×3 , so as to obtain fused multi-scale features; the fused multi-scale features and the features captured by the point-level convolution kernel are elementally added, so as to obtain a new spliced feature; the new spliced feature is subjected to a convolution operation again, so as to obtain an output feature O fused according to the weight self-selection of the different scale features:

[0092] O=Conv{x0+x1×S[:,0]+x2×S[:,1]+x3×S[:,2]} (4)

[0093] An adaptive fusion module is configured to perform adaptive fusion on the different features extracted in the channel dimension by an SE module based on the output feature O, expand a receptive field and perform three-dimensional spatial feature fusion, so as to obtain a feature map after adaptive feature selection.

[0094] A calculation module is configured to perform one-time weighted addition on the original feature map and the feature map after adaptive feature selection, and take a weighted addition result as a final output according to a preset weighting coefficient.

[0095] In some embodiments, the acquisition module is further configured to:

[0096] The small, medium and large scale feature maps x1, x2 and x3 are output as the output of the multi-scale feature extraction module and are transmitted to the subsequent feature fusion module.

[0097] f i :X→U∈R N×4 (i=1,2,3) (1)

[0098] Each f i is a combination of different Convolutional layer, Activation Function layer and Batch Normalization layer.

[0099] In some embodiments, the adaptive fusion module is further configured:

[0100] The output feature O is subjected to global average pooling, then the output 1x1xC data is subjected to two-level full connection operation, finally, the value is limited to the range of [0, 1] by using a Sigmoid activation function, and then the value is multiplied by scale to the C channels of O as the input feature of the next stage; by controlling the size of scale, important features are enhanced, and features that contribute to the final result are adaptively selected, and unimportant features are weakened, thereby enhancing the directivity of the extracted features.

[0101] It should be noted that the modules described in the embodiments of the present application can be implemented in software or hardware, and the described modules can be arranged in a processor. In some cases, the name of the module does not constitute a limitation on the module itself.

[0102] The point cloud segmentation device based on multi-scale feature extraction and fusion mentioned in the embodiments of the present application and the method described above belong to the same technical concept, and have basically the same technical effects, which will not be described here.

[0103] The embodiments of the present application also provide a non-transitory computer readable medium storing instructions, when the instructions are executed by a processor, the method according to any one of the embodiments of the present application is executed.

[0104] Furthermore, although example embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of

[0105] The foregoing description is intended to be illustrative and not exclusive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. Other examples can be utilized, as would be understood by one of ordinary skill in the art upon reading the above description. The various features can also be grouped for simplicity and to vary examples of the application. This should not be interpreted as a requirement that the features sited belon to any claim— a claim can include or exclude all these features. Rather, inventive subject matter can exist in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the description of the application by this reference, with each claim standing on its own as a separate embodiment, and the scope of the application should be afforded the full breadth of each claim, and equivalents thereof.

Claims

1. A point cloud segmentation method based on multi-scale feature extraction and fusion, characterized in that, The method comprises: According to different scale features, the acquired point cloud data is captured by using four scale convolution kernels of point level, small scale, medium scale and large scale respectively to obtain four feature map outputs of different scales, which are x0, x1, x2 and x3 respectively; The small scale, medium scale and large scale feature map outputs are spliced together and input into a Softmax activation function to obtain a spliced feature map: U cat = cat(U1,U2,U3) (2) In the formula, U cat ∈R N×3 , R represents a domain, N represents a point number, 3 is a feature dimension, and the attention weight represented by different scales obtained is represented as: S = softmax(U cat ) (3) The features of different scales are fused according to S∈R N×3 The element addition is performed to obtain the fused multi-scale features; the element addition is performed between the fused multi-scale features and the features captured by the point-level convolution kernel to obtain a new spliced feature, and the new spliced feature is subjected to a convolution operation to obtain the output feature O after the self-selection fusion of the features of different scales according to the weight. O=Conv{x0+x1×S[:,0]+x2×S[:,1]+x3×S[:,2]} (4) Based on the output feature O, different features extracted are adaptively fused in the channel dimension by an SE module, the receptive field is expanded, three-dimensional spatial feature fusion is performed, and a feature map after adaptive feature selection is obtained. The original feature map and the feature map after adaptive feature selection are added by weighting, and the weighted addition result is taken as the final output according to a preset weighting coefficient.

2. The method of claim 1, wherein, The small scale convolution kernel is set to 1*1*1, the medium scale convolution kernel is set to 2*2*2, the large scale convolution kernel is set to 4*4*4, and the point level convolution kernel is set to a sub-manifold convolution, and the convolution kernel size of the sub-manifold convolution is 1 and the convolution step is set to 1.

3. The method of claim 1, wherein, The method further comprises: The small, medium and large scale feature map outputs x1, x2 and x3 are taken as the output of the multi-scale feature extraction module and are transmitted to the subsequent feature fusion module, in order to avoid using the same weight for different scale features, the features of large, medium and small scales are added to obtain an intermediate feature map X, and then three different multi-layer perceptrons are used to transform the intermediate feature map X: f i : X→ U∈R N×4 (i = 1, 2, 3) (1) Each of the f in the formula i is a combination of different convolutional layers, activation function layers, and batch normalization layers.

4. The method of claim 1, wherein, Based on the output feature O, different features extracted are adaptively fused in the channel dimension by an SE module, the receptive field is expanded, three-dimensional spatial feature fusion is performed, and a feature map after adaptive feature selection is obtained, comprising: The output feature O is globally averaged pooled, then the output 1*1*C data is further operated by two-level full connection, finally limited to the range of [0, 1] by using a Sigmoid activation function, and then the value is multiplied by C channels of O as the input feature of the next level; By controlling the size of scale, important features are enhanced, features that contribute to the final result are adaptively selected, and unimportant features are weakened, so as to enhance the directivity of the extracted features.

5. The method of claim 4, wherein, The calculation process of globally averaging the output feature O, then operating the output 1*1*C data by two-level full connection, and finally limiting to the range of [0, 1] by using a Sigmoid activation function is as follows: where δ represents the ReLU activation function, σ represents the SigMoid activation function, u c (i,j) represents the features of the input feature map at any point, H and L represent the height and width of the feature map, W represents a linear transformation, and represent two linear transformations, and the number of feature channels is compressed from C to to reduce the model complexity, where r is a dimension reduction coefficient; in order to restore the number of compressed channels in the first step to the original number of channels, a second linear transformation is performed, and finally the scale is controlled. According to the attention weight calculated according to formula (3), the features of different channels are adaptively fused.

6. The method of claim 5, wherein, The weighting coefficient is set to 0.

4.

7. A point cloud segmentation device based on multi-scale feature extraction and fusion, characterized in that, The device comprises: An acquisition module configured to capture features of point cloud data acquired according to different scale features by using four scale convolution kernels of point level, small scale, medium scale and large scale respectively to obtain four feature map outputs of different scales, which are x0, x1, x2 and x3 respectively; The fusion module is configured to splice the small-scale, medium-scale and large-scale feature maps together and send them into a Softmax activation function to obtain a spliced feature map: U cat = cat(U1,U2,U3) (2) where U cat ∈R N×3 , R represents a domain, N represents a point number, 3 is a feature dimension, and the attention weight represented by different scales obtained is represented as: S = softmax(U cat ) (3) The features of different scales are fused according to S∈R N×3 The element addition is performed to obtain the fused multi-scale features; the element addition is performed between the fused multi-scale features and the features captured by the point-level convolution kernel to obtain a new spliced feature, and the new spliced feature is subjected to a convolution operation to obtain the output feature O after the self-selection fusion of the features of different scales according to the weight. O = Conv{x0+x1×S[:,0]+x2×S[:,1]+x3×S[:,2]} (4) The adaptive fusion module is configured to perform adaptive fusion of the extracted different features in a channel dimension based on the output feature O through an SE module, expand a receptive field, and perform three-dimensional spatial feature fusion to obtain a feature map after adaptive feature selection. The calculation module is configured to perform one-time weighted addition on the original feature map and the feature map after adaptive feature selection, and take a weighted addition result as a final output according to a preset weighting coefficient.

8. The apparatus of claim 7, wherein, The acquisition module is further configured to: The small-scale, medium-scale and large-scale feature map outputs x1, x2 and x3 are delivered to the subsequent feature fusion module as outputs of the multi-scale feature extraction module, so as to avoid using the same weight for different scale features, add the large-scale, medium-scale and small-scale features to obtain an intermediate feature map X, and then use three different multi-layer perceptrons to transform the intermediate feature map X: f i : X→ U∈R N×4 (i = 1, 2, 3) (1) Each of the f in the formula i is a combination of different convolutional layers, activation function layers, and batch normalization layers.

9. The apparatus of claim 8, wherein, The adaptive fusion module is further configured: Global average pooling is performed on the output feature O, then two-level fully connected operations are performed on the output 1x1xC data, finally a Sigmoid activation function is used to limit the value to a range of [0, 1], and then the value is multiplied by C channels of O as an input feature of the next stage; By controlling the size of the scale, important features are enhanced, adaptive selection of features that contribute to the final result is performed, unimportant features are weakened, and the directionality of the extracted features is enhanced. 10.A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, perform the method according to any one of claims 1 to 6.

Images