Three-dimensional semantic segmentation method for substation scenes based on convolution and attention

Abstract

The present invention discloses a three-dimensional semantic segmentation method for a substation scene based on a combination of convolution and attention, comprising the following steps: S1, dividing the three-dimensional point cloud data in the substation scene into regions and categories, setting semantic labels for different categories, dividing the training set and the test set according to the regions, and forming a three-dimensional semantic segmentation data set; S2, constructing a three-dimensional semantic segmentation model based on a convolution and attention mechanism combined module to predict the semantic labels of point cloud data in the substation data set; S3, determining a loss function and a parameter optimizer, and training the three-dimensional semantic segmentation model in combination with the three-dimensional semantic segmentation data set; S4, semantically segmenting the point cloud data in the substation scene based on the trained three-dimensional semantic segmentation model. Compared with the existing three-dimensional semantic segmentation method, the present invention can effectively improve the accuracy of three-dimensional semantic segmentation and realize accurate semantic segmentation of point cloud data in the substation scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of substation scene point cloud data processing, and specifically relates to a three-dimensional semantic segmentation method for substation scenes based on the combination of convolution and attention. Background Art

[0002] With the rapid development of high-precision sensors such as LiDAR and Kinect, 3D point cloud data has become the main data format for representing the real world. After the sensor obtains the 3D point cloud data in the scene, it is necessary to have a deep understanding of the 3D scene in order to perform accurate object recognition and scene analysis. Therefore, a 3D semantic segmentation algorithm is needed to learn to recognize large 3D scenes. 3D semantic segmentation is a fundamental problem in computer vision and computer graphics. It is a process of learning to perceive 3D point cloud data. The goal of 3D semantic segmentation is to establish a mapping function to predict the fine-grained labels of objects in the 3D scene, so as to understand the semantic meaning of each point in the scene. 3D semantic segmentation has a wide range of applications, such as autonomous driving, mobile robots, industrial control, augmented reality, and medical image analysis. Since 3D semantic segmentation technology plays a vital role in applications such as autonomous driving, mobile robots, industrial control, augmented reality, and medical image analysis, there is an urgent need to conduct more in-depth research on 3D semantic segmentation technology to benefit these applications.

[0003] In recent years, people have conducted in-depth and extensive research on 3D semantic segmentation methods based on deep learning. Currently, 3D semantic segmentation methods based on deep learning can be divided into projection-based methods, voxel-based methods, and point-based methods. Point-based methods include convolution-based methods and Transformer-based methods. Both methods have shown excellent performance in the semantic segmentation task of point cloud data. Convolution-based methods show strong performance in extracting local features of the model through convolution operations, and Transformer-based methods can better extract the global context information of point clouds through attention mechanisms. However, many existing methods still have the problem of insufficient accuracy when facing complex point cloud data in substation scenarios. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a three-dimensional semantic segmentation method for substation scenes based on the combination of convolution and attention. Compared with the existing three-dimensional semantic segmentation method, the accuracy of three-dimensional semantic segmentation can be effectively improved, and accurate semantic segmentation of substation scene point cloud data can be achieved.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] The 3D semantic segmentation method of substation scene based on the combination of convolution and attention includes the following steps:

[0007] S1. Divide the 3D point cloud data in the substation scene into regions and categories, set semantic labels for different categories, divide the training set and test set into regions, and form a 3D semantic segmentation dataset;

[0008] S2, build a 3D semantic segmentation model based on the convolution and attention mechanism combined module to predict the semantic labels of point cloud data in the substation dataset;

[0009] S3, determining a loss function and a parameter optimizer, and training a 3D semantic segmentation model in combination with a 3D semantic segmentation dataset;

[0010] S4. Perform semantic segmentation on point cloud data in the substation scene based on the trained 3D semantic segmentation model.

[0011] To optimize the above technical solutions, the specific measures taken also include:

[0012] The above-mentioned S1 specifically includes the following steps:

[0013] For all point cloud data except substation equipment, the data are classified according to the object's form, function and distribution information, including wires, buildings and ground. According to the scale of substation equipment and the number of point clouds contained in the object, the substation equipment point cloud data category is divided into small equipment, medium equipment and large equipment. Semantic labels are set for different categories.

[0014] The dataset is divided into regions according to the spatial information and feature information of the point cloud data. The substation cloud data is divided into regions of different sizes, and the training set and test set are divided according to the regions. The test set includes multiple blocks of different sizes and contains all categories.

[0015] The above-mentioned S2 uses the PointNeXt network as the backbone network for feature learning of point cloud data to construct a three-dimensional semantic segmentation model; the backbone network includes an MLP layer and four downsampling modules, each downsampling module includes a convolution and attention mechanism combination module and an inverted bottleneck MLP module in PointNeXt; wherein, the convolution and attention mechanism combination module is used as a feature extraction module to extract local and global features of point cloud data; the inverted bottleneck MLP module is used to enrich the obtained features.

[0016] The process of extracting features by combining the above-mentioned convolution and attention mechanisms with the module is as follows: sampling and grouping the input point cloud: selecting the center point by the farthest point sampling method, finding the neighborhood of each center point by the ball query algorithm, and dividing the input point cloud into several neighborhoods of the same size; extracting the local features lf of each neighborhood by the local feature extraction module LFE; inputting the local features lf, the features nf of the neighborhood points, and the relative positions pr of the neighborhood points relative to the center point into the global feature extraction module GFE to further extract the global information and obtain the global features gf; adding the local features lf and the global features gf element by element to obtain the final output feature value f.

[0017] The specific operation of the above-mentioned local feature extraction module LFE is: for points in the neighborhood, firstly, local features are extracted through convolution operation, and then the features are aggregated to the center points of each neighborhood through maximum pooling operation to obtain local features lf.

[0018] The above-mentioned global feature extraction module GFE includes a linear layer and a vector attention module.

[0019] The encoding process of the above vector attention module is as follows:

[0020] ;

[0021] Among them, VA represents the vector attention mechanism; lf is the feature value of the center point after the feature aggregation of the local feature extraction module LFE; nf is the feature value of the neighborhood point found by the ball query algorithm with lf as the center; pr represents the relative position between the neighborhood point and the center point in the neighborhood;

[0022] Linear represents the linear layer; Q, K, V, and pe are query matrix, key matrix, value matrix, and position encoding, respectively;

[0023] Softmax( , dim=-2) means normalizing the input value in the second-to-last dimension;

[0024] gf represents the global feature obtained after the vector attention mechanism.

[0025] The loss function of S3 mentioned above adopts the cross entropy loss function, and the formula is:

[0026] ;

[0027] Where n represents the number of categories; and The original output logits and the true category label target are converted into probability distributions through the softmax function.

[0028] The parameter optimizer mentioned above is the AdamW optimizer, with a learning rate of 0.01 and a weight decay rate of .

[0029] The present invention has the following beneficial effects:

[0030] The present invention combines convolution and attention mechanisms to enable the model to take advantage of the two methods, improve the model's ability to extract local and global features, enhance the accuracy of semantic segmentation of substation data, and solve the technical problem of insufficient accuracy of existing three-dimensional semantic segmentation methods when facing complex point cloud data in substation scenarios.

[0031] The three-dimensional semantic segmentation model of the present invention adopts PointNeXt network as the backbone network for feature learning of point cloud data; and adds a vector attention mechanism for extracting global context information of point cloud data in the downsampling layer of the backbone network. In the process of generating attention weights and obtaining final features, position encoding is added; the three-dimensional semantic segmentation model improves the ability of the model to extract local features and global features of point cloud data by constructing a suitable convolution and attention mechanism combination module. In the local neighborhood obtained by the ball query algorithm with the sampling point as the center point, the calculation and memory cost of the model are reduced.

[0032] The encoding process of the vector attention module of the present invention, based on the original vector attention mechanism, innovatively uses cross-vector attention in the three-dimensional semantic segmentation task, so that the global feature extraction module GFE can take the local feature lf and the corresponding original neighborhood feature nf as input, while further extracting the global context information, and realize the information interaction between the local feature extraction module LFE and the global feature extraction module GFE, thereby effectively improving the model's ability to extract local features and global features. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 is a flow chart of the three-dimensional semantic segmentation method of the present invention;

[0035] Figure 2 This is a schematic diagram of the vector attention mechanism after adding position encoding according to the present invention;

[0036] Figure 3 This is a schematic diagram of the feature extraction module after combining convolution and attention mechanism in the present invention;

[0037] Figure 4 This is a schematic diagram of the three-dimensional semantic segmentation network framework proposed by the present invention;

[0038] Figure 5 It is a schematic diagram of the three-dimensional semantic segmentation results of the point cloud data in the substation dataset of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0041] Based on the technical defects of existing technologies when facing complex point cloud data in substation scenarios, please refer to Figure 1-Figure 5 , the embodiment of the present invention proposes a three-dimensional semantic segmentation method for substation scenes based on the combination of convolution and attention mechanism. Specifically, Figure 1 An optional flow chart of the method is shown as follows: Figure 1 As shown, the method comprises the following steps:

[0042] S1. Divide the 3D point cloud data in the substation scene into regions and categories, set semantic labels for different categories, divide the training set and test set into regions, and form a 3D semantic segmentation dataset;

[0043] As a preferred implementation of step S1, constructing a 3D semantic segmentation dataset based on the 3D point cloud data in the substation scene specifically includes the following steps:

[0044] S11, dividing the data set into regions according to the spatial information and feature information of the point cloud data; as a preferred implementation of step S11, specifically comprising the following steps:

[0045] S111. Divide the substation cloud data into two parts according to the deep learning data set requirements, one part as a training set and the other part as a test set;

[0046] S112, dividing the data set into regions according to the spatial information and feature information of the point cloud data, and dividing the substation cloud data into 6 regions of different sizes. Among them, region 5 is used as a test set, which contains 13 blocks of different sizes and all categories; the other regions are used as training sets, which contain a total of 29 blocks of different sizes and all categories.

[0047] S12, classifying the data according to the different forms of objects existing in the substation scene; as a preferred implementation of step S12, specifically including the following steps:

[0048] S121, firstly, classify all point cloud data except substation equipment into seven categories, namely, wires, buildings, and ground, according to the form, function, distribution and other information of the objects;

[0049] S122. Divide the substation equipment into small equipment, medium equipment and large equipment according to the scale of the substation equipment and the number of point clouds contained in the object.

[0050] S13, according to the designed data set structure, complete the manual segmentation of the substation cloud data through the point cloud processing software, and add semantic labels to different objects obtained by segmentation according to the designed data set categories;

[0051] S14. Perform semantic segmentation on the substation dataset using the existing three-dimensional semantic segmentation method to verify the universality of the substation dataset in the semantic segmentation task.

[0052] S2, build a 3D semantic segmentation model based on the convolution and attention mechanism combined module to predict the semantic labels of point cloud data in the substation dataset;

[0053] As a preferred implementation of step S2, a three-dimensional semantic segmentation model is constructed to predict the semantic labels of point cloud data in the substation data set; specifically, the following steps are included:

[0054] S21, using PointNeXt network as the backbone network for feature learning of point cloud data;

[0055] S22. Add a vector attention mechanism to extract global context information from point cloud data in the downsampling layer of the backbone network, such as Figure 2 As shown; the encoding process of the vector attention mechanism is as follows:

[0056] ;

[0057] Among them, VA represents the vector attention mechanism; lf is the feature value of the center point after the feature aggregation of the local feature extraction module LFE; nf is the feature value of the neighborhood point found by the ball query algorithm with lf as the center; pr represents the relative position between the neighborhood point and the center point in the neighborhood;

[0058] Linear represents a linear layer;

[0059] lf and nf are processed by different linear layers to obtain the query matrix Q, key matrix K and value matrix V required by the attention mechanism, and the relative position pr is processed to obtain the position encoding pe. Softmax(, dim=-2) means normalizing the input value on the second-to-last dimension. gf represents the global feature obtained after the vector attention mechanism.

[0060] As shown in the above encoding process, based on the original vector attention mechanism, the present invention innovatively uses cross-vector attention in the three-dimensional semantic segmentation task, so that the global feature extraction module GFE can take the local feature lf and the corresponding original neighborhood feature nf as input, while further extracting the global context information, and realize the information interaction between the local feature extraction module LFE and the global feature extraction module GFE, thereby effectively improving the model's ability to extract local features and global features.

[0061] S23. Construct a suitable module combining convolution and attention mechanism to improve the model's ability to extract local and global features of point cloud data, such as Figure 3 As shown; as a preferred implementation of step S23, specifically comprising the following steps:

[0062] S231. For the input point cloud, it is first sampled and grouped, that is, the center point is selected by the farthest point sampling method. After the center point is determined, the neighborhood of each center point is found by the ball query algorithm.

[0063] S232. After dividing the input point cloud into several neighborhoods of the same size, the local features of each neighborhood are extracted through the local feature extraction module (LFE). The specific operation of the local feature extraction module is as follows: for the points in the neighborhood, the local features are first extracted through the convolution operation, and then the features are aggregated to the center points of each neighborhood through the MaxPooling operation to obtain the local features lf.

[0064] S233, after obtaining the local feature lf, lf, the feature nf of the neighborhood point and the relative position pr of the neighborhood point relative to the center point are passed to the global feature extraction module (GFE) to further extract global information to obtain the global feature gf. The global feature extraction module is composed of a linear layer, a vector attention module and another linear layer.

[0065] S234: After obtaining the local feature lf and the global feature gf, the two are added element by element to obtain the final output feature value f.

[0066] S24. Design a network framework for the downsampling layer of the backbone network that combines convolution and attention mechanisms, and determine the input and output of the backbone network.

[0067] like Figure 4 As shown, the backbone network consists of an MLP layer and four downsampling modules, each of which consists of a convolution and attention mechanism combined module and an inverted bottleneck MLP module in PointNeXt, wherein the convolution and attention mechanism combined module is used to extract local and global features of point cloud data, and the inverted bottleneck MLP module is used to enrich the obtained features;

[0068] And, the input of the entire backbone network is divided into two parts. One part is the spatial information of the point cloud data, represented by P, which contains the spatial coordinates x, y, z of the point cloud data, and its dimension is (B, N, 3), where B represents the batch size and N represents the number of points; the other part is the feature information of the point cloud, represented by F, which contains the color information r, g, b of the point cloud and the height information h of the point cloud, and its dimension is (B, N, 4);

[0069] Spatial information P is used for position embedding, and feature information F is passed to the backbone network to further extract global and local features. After the backbone network feature learning, the feature value is obtained. , whose dimension is (B, N / 256, 1024). When performing semantic segmentation on the substation dataset, the value of N is set to 4096.

[0070] S3, determining a loss function and a parameter optimizer, and training a 3D semantic segmentation model in combination with a 3D semantic segmentation dataset;

[0071] As a preferred implementation of step S3, the loss function and optimization parameters used to improve the accuracy of the three-dimensional semantic segmentation model in predicting the point cloud semantic labels are determined as follows:

[0072] First, the original output logits of the model and the label target of the real category are converted into probability distributions through the softmax function. and , the softmax function is calculated as follows:

[0073] ;

[0074] in, Represents the i-th input value, and n is the length of the tensor, that is, the number of input point clouds.

[0075] After that, we use the true category label target and the model output logits probability distribution to calculate the cross entropy loss. Specifically, we find the predicted probability corresponding to the true category and calculate its negative logarithm. The formula is:

[0076] ;

[0077] Here, n represents the number of categories.

[0078] In addition, in this embodiment, the AdamW optimizer is selected to update the parameters of the semantic segmentation model, where the learning rate of the AdamW optimizer is 0.01 and the weight decay rate is , iteratively update the model parameters until the loss function of the semantic segmentation model Until convergence.

[0079] Experimental verification shows that such settings of optimizer parameters can enable the model to achieve the best prediction effect.

[0080] S4. Based on the trained 3D semantic segmentation model, accurate semantic segmentation of point cloud data in the substation scene is achieved.

[0081] Based on the above algorithm, the semantic segmentation results of the substation dataset are as follows: Figure 5 As shown, Figure 5 (a) shows the real label of the original point cloud in the substation scene; after being processed by the three-dimensional semantic segmentation method proposed in this embodiment, the prediction result is as follows Figure 5 Therefore, this embodiment solves the technical problem that the existing 3D semantic segmentation method is not accurate enough when facing complex point cloud data in the substation scene, and realizes accurate semantic segmentation of substation site cloud data.

[0082] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0083] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A 3D semantic segmentation method for substation scenes based on the combination of convolution and attention, characterized in that: The following steps are involved: S1. Divide the 3D point cloud data in the substation scene into regions and categories, set semantic labels for different categories, divide the training set and test set into regions, and form a 3D semantic segmentation dataset; S2. Construct a 3D semantic segmentation model based on the convolution and attention mechanism combined module to predict the semantic labels of point cloud data in the substation dataset; specifically: A 3D semantic segmentation model is constructed by using the PointNeXt network as a backbone network for feature learning of point cloud data; the backbone network comprises an MLP layer and four downsampling modules, each of which comprises a convolution and attention mechanism combination module and an inverted bottleneck MLP module in PointNeXt; wherein the convolution and attention mechanism combination module is used as a feature extraction module to extract local and global features of point cloud data; and the inverted bottleneck MLP module is used to enrich the obtained features; The process of extracting features by combining the convolution and attention mechanisms is as follows: sampling and grouping the input point cloud: selecting the center point by the farthest point sampling method, finding the neighborhood of each center point by the ball query algorithm, and dividing the input point cloud into several neighborhoods of the same size; extracting the local features lf of each neighborhood by the local feature extraction module LFE; inputting the local features lf, the features nf of the neighborhood points, and the relative positions pr of the neighborhood points relative to the center point into the global feature extraction module GFE to further extract global information and obtain the global features gf; adding the local features lf and the global features gf element by element to obtain the final output feature value f; S3, determining a loss function and a parameter optimizer, and training a 3D semantic segmentation model in combination with a 3D semantic segmentation dataset; S4. Perform semantic segmentation on point cloud data in the substation scene based on the trained 3D semantic segmentation model.

2. According to claim 1, the three-dimensional semantic segmentation method of substation scene based on convolution and attention is characterized in that: The S1 specifically includes the following steps: For all point cloud data except substation equipment, the data are classified according to the object's form, function and distribution information, including wires, buildings and ground. According to the scale of substation equipment and the number of point clouds contained in the object, the substation equipment point cloud data category is divided into small equipment, medium equipment and large equipment. Semantic labels are set for different categories. The dataset is divided into regions according to the spatial information and feature information of the point cloud data. The substation cloud data is divided into regions of different sizes, and the training set and test set are divided according to the regions. The test set includes multiple blocks of different sizes and contains all categories.

3. According to claim 1, the three-dimensional semantic segmentation method of substation scene based on convolution and attention is characterized in that: The specific operation of the local feature extraction module LFE is as follows: for points in the neighborhood, firstly, local features are extracted through a convolution operation, and then the features are aggregated to the center points of each neighborhood through a maximum pooling operation to obtain local features lf.

4. The three-dimensional semantic segmentation method of substation scene based on convolution and attention according to claim 1 is characterized in that: The global feature extraction module GFE includes a linear layer and a vector attention module.

5. According to claim 4, the method for 3D semantic segmentation of substation scenes based on the combination of convolution and attention is characterized in that: The encoding process of the vector attention module is as follows: ； Among them, VA represents the vector attention mechanism; lf is the feature value of the center point after the feature aggregation of the local feature extraction module LFE; nf is the feature value of the neighborhood point found by the ball query algorithm with lf as the center; pr represents the relative position between the neighborhood point and the center point in the neighborhood; Linear represents the linear layer; Q, K, V, and pe are query matrix, key matrix, value matrix, and position encoding, respectively; Softmax( , dim=-2) means normalizing the input value in the second-to-last dimension; gf represents the global feature obtained after the vector attention mechanism.

6. The three-dimensional semantic segmentation method of substation scene based on convolution and attention according to claim 1 is characterized in that: The loss function of S3 adopts the cross entropy loss function, and the formula is: ； Where n represents the number of categories; and The original output logits and the true category label target are converted into probability distributions through the softmax function.

7. The three-dimensional semantic segmentation method of substation scene based on convolution and attention according to claim 1 is characterized in that: The parameter optimizer is the AdamW optimizer, with a learning rate of 0.01 and a weight decay rate of .

Images