Semantic segmentation method and system for lidar point cloud of high-voltage transmission line

Abstract

The present invention provides a method and system for semantic segmentation of lidar point clouds of high-voltage transmission lines. The implementation scheme is as follows: based on a training set of high-voltage transmission line point cloud data, a first network structure constructed by a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module is trained to obtain a target semantic segmentation model; in response to a semantic segmentation request, semantic segmentation is performed on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model. Wherein, the first network structure includes an encoder and a decoder; the encoder and the decoder include at least one first encoding layer and a first decoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel, and at least one second encoding layer and a second decoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel and then connected in series with a channel and spatial attention module. By adopting the present invention, the accuracy of semantic segmentation of point cloud data of high-voltage transmission lines can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a semantic segmentation method and system for high-voltage transmission line laser radar point clouds. Background Art

[0002] High-voltage transmission line inspections are a daily routine task for power systems. They are crucial for the safe and stable operation of power supply. The primary purpose of high-voltage transmission line inspections is to verify the integrity of power infrastructure, including power lines, towers, and hardware. They also check for potential hazards such as tree obstructions, geological hazards, and tower tilt.

[0003] In recent years, with the widespread adoption of observation technology, the use of remote sensing, particularly lidar (LiDAR), to observe transmission lines has become a mainstream inspection trend. LiDAR employs an active observation method, can be deployed on multiple platforms, and offers high precision, high speed, and minimal weather impact. Consequently, the use of LiDAR systems mounted on manned or unmanned aircraft is increasingly becoming a key data collection method for high-voltage transmission line inspections in the power industry. Furthermore, extracting scene information about high-voltage transmission lines based on LiDAR 3D point cloud data has become a crucial component of inspections.

[0004] Some technologies offer semantic segmentation models for power transmission corridors, such as the enhanced PointNet model, which combines a geometric feature extraction module with a neighborhood information aggregation module to segment railway power lines and towers from railway scene point clouds. Another example is the PointNet++ model, which performs high-precision segmentation of power lines and towers. The Coordinate Attention (CA) module is then integrated with PointNet++ to implement an end-to-end CA-PointNet++ model. However, several challenges remain for semantic segmentation of airborne LiDAR in power transmission corridors:

[0005] 1. Airborne lidar has a larger geographical range. The above methods mainly focus on how to express the local neighborhood information of points, which makes it difficult to encode global context information for large-scale point cloud scenes.

[0006] 2. The transmission channel scenario is a long strip extending continuously along the transmission line, with a certain length as a buffer zone. Transmission channels are characterized by large scale, large point cloud data volumes (the number of points between two towers can reach over 10 million), and significant category diversity (for example, important elements such as power towers and power lines account for a small proportion). Therefore, conventional point cloud semantic segmentation methods are not adequate for understanding transmission channel scenarios. Summary of the Invention

[0007] The present invention provides a semantic segmentation method and system for high-voltage transmission line lidar point cloud, which can solve at least one of the above technical problems.

[0008] According to one aspect of the present invention, a semantic segmentation method for a high-voltage transmission line lidar point cloud is provided, comprising:

[0009] The first network structure is constructed based on the kernel point convolution layer, the graph edge convolution layer, and the channel and spatial attention modules;

[0010] Based on the high-voltage transmission line point cloud data training set, the first network structure is trained to obtain a target semantic segmentation model;

[0011] In response to the semantic segmentation request, performing semantic segmentation on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model;

[0012] Wherein, the first network structure includes an encoder and a decoder;

[0013] The encoder includes at least one first encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel, and at least one second encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel and then connected in series with the channel and spatial attention module;

[0014] The decoder includes at least one first decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module.

[0015] According to another aspect of the present invention, a semantic segmentation device for a high-voltage transmission line laser radar point cloud is provided, the device comprising:

[0016] A network structure determination module, configured to construct a first network structure based on a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module;

[0017] A model training module, configured to train the first network structure based on a training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model;

[0018] a semantic segmentation module, configured to respond to a semantic segmentation request and perform semantic segmentation on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model;

[0019] Wherein, the first network structure includes an encoder and a decoder;

[0020] The encoder includes at least one first encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel, and at least one second encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel and then connected in series with the channel and spatial attention module;

[0021] The decoder includes at least one first decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module.

[0022] The technical solution of the present invention is adopted to construct a first network structure based on a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module. The first network structure is trained based on a training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model. In addition, the first network structure includes an encoder and a decoder. The encoder includes at least one first encoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel, and at least one second encoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel, followed by a channel and spatial attention module in series. The decoder includes at least one first decoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel, and at least one second decoding layer composed of a kernel point convolution layer and a graph edge convolution layer in parallel, followed by a channel and spatial attention module in series. In this way, the target semantic segmentation model can use the kernel point convolution layer and the graph edge convolution layer to perform local and global feature fusion on the point cloud data respectively during encoding and decoding, and can use the channel and spatial attention modules to understand the context of the fused features during encoding and decoding. Finally, the fused features after context understanding obtained by decoding are used to realize the semantic segmentation of the point cloud data of the high-voltage transmission line, thereby improving the accuracy of semantic segmentation.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention.

[0025] Figure 1 This is a flow chart of a semantic segmentation method for a high-voltage transmission line lidar point cloud according to an embodiment of the present invention;

[0026] Figure 2 1 is a network structure diagram of a graph edge convolution layer according to an embodiment of the present invention;

[0027] Figure 3 is a network structure diagram of the channel and spatial attention module according to one embodiment of the present invention;

[0028] Figure 4 is a schematic structural diagram of the network layer according to an embodiment of the present invention;

[0029] Figure 5 is a schematic diagram of a first network structure according to an embodiment of the present invention;

[0030] Figure 6 This is a structural block diagram of a semantic segmentation device for a high-voltage transmission line lidar point cloud according to an embodiment of the present invention;

[0031] Figure 7 is a block diagram of an electronic device for implementing the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, and various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] Figure 1 This is a flowchart of a semantic segmentation method for a high-voltage transmission line lidar point cloud according to an embodiment of the present invention.

[0034] like Figure 1 As shown, the semantic segmentation method of the high-voltage transmission line lidar point cloud may include:

[0035] S110, constructing the first network structure based on the kernel point convolution layer, the graph edge convolution layer, and the channel and spatial attention modules;

[0036] S120, training the first network structure based on the high-voltage transmission line point cloud data training set to obtain a target semantic segmentation model;

[0037] S130, in response to the semantic segmentation request, performing semantic segmentation on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model;

[0038] Wherein, the first network structure includes an encoder and a decoder;

[0039] The encoder includes at least one first encoding layer consisting of a kernel point convolution layer and a graph edge convolution layer in parallel, and at least one second encoding layer consisting of a kernel point convolution layer and a graph edge convolution layer in parallel followed by a channel and spatial attention module in series.

[0040] The decoder includes at least one first decoding layer consisting of a kernel point convolution layer and a graph edge convolution layer in parallel, and at least one second decoding layer consisting of a kernel point convolution layer and a graph edge convolution layer in parallel followed by a series channel and spatial attention module.

[0041] For example, in the kernel point convolution layer, each convolution kernel consists of a set of kernel points with fixed coordinates in three-dimensional space. The kernel point convolution layer calculates the weights based on the relative positions between each point in the input point cloud data and the kernel points, thereby generating a convolution result and outputting the convolution result. The input data of the kernel point convolution layer includes point position information and point feature information. Point position information is usually expressed as The coordinate representation is used to describe the specific position of each point in three-dimensional space. These coordinates are used to determine the spatial relationship and structural information of each point in the point cloud. The kernel point convolution layer uses a set of kernel points To define the convolution kernel. Each kernel point These kernel points have fixed coordinates in three-dimensional space. These kernel points are similar to the filter weights in traditional convolution, but are irregularly distributed in three-dimensional space. The positions of these kernel points are adjusted during training to optimize the extracted features.

[0042] For example, the graph edge convolution layer can construct a graph on a line segment composed of geometrically homogeneous points to capture the relationship between objects. By combining segment features and point features, the network can adaptively encode local and global features, thereby achieving better semantic learning or prediction on airborne lidar datasets. Figure 2 As shown, for edge conditioned convolution (ECC), it is considered as a directed graph or an undirected graph ,in, is a finite set of vertices, ; is a set of edges, .set up is the layer index in the feedforward neural network. Thus, we define a function to assign labels to each vertex, and a function to assign labels to each edge.

[0043] like Figure 2 As shown in Figure 1, an unsupervised algorithm is used to segment the point cloud in the directed or undirected graph into multiple independent point cloud clusters. This is pre-computed before training using semantic labels and pre-set geometric features. This ensures that the structure of the directed or undirected graph remains fixed during training, eliminating the need to repeatedly update the labels of each edge in the graph and reducing unnecessary computational overhead.

[0044] Then, we use the edge condition to convolve the graph structure and dynamically generate filter weights to flexibly handle neighborhood relationships. Specifically, given a point cloud and its geometric characteristics (e.g. linearity or normal vector), construct a directed graph , and set the labels of each vertex in the directed graph as follows For point clouds Each point in , create a vertex for it and through Determine its label. Then, each vertex All vertices in its spatial neighborhood Connected by directed edges. Edge Conditional Convolution (ECC) can dynamically generate filter weights based on directed edges and flexibly handle the number of neighbors, thereby capturing contextual information between different edges.

[0045] For example, Figure 3 As shown in FIG, the channel and spatial attention module can be composed of a channel attention network and a spatial attention network connected in parallel and then connected to a connection layer.

[0046] For example, Figure 3 As shown in the figure, the channel attention network projects input features into different feature subspaces through different learnable fully connected layers to construct queries, keys, and values ​​in the attention function. The attention function has the descriptive power to encode global context, and its output is enhanced point cloud features. The channel attention mechanism focuses on selecting and enhancing the most effective features for the network. By studying the relationships between features, it determines the weights of different features and multiplies these weights by the features to obtain the features that are important for network classification. The weight calculation process can compress the spatial dimensions of the input features, improving computational efficiency.

[0047] For the channel attention network, average pooling and max pooling are used to aggregate spatial dimensional information across different feature channels, resulting in average and max pooled features. These two features are then fed into a multi-layer perceptron (MLP) network to generate mapping functions for important features. The MLP network can consist of two fully connected layers, an activation function, and a random dropout mechanism for the output of the second fully connected layer, which can improve the network's generalization capabilities. Finally, the output of the channel attention network is obtained by element-wise summing the results of the two different features processed by the MLP.

[0048] like Figure 3As shown in Figure 1, the spatial attention network is used to select neighborhoods that are more conducive to expressing point cloud shape information. To prioritize the correlations between different point cloud categories, the input features are first average pooled and max pooled. The input features are a feature matrix that aggregates graph edge conditional convolution and kernel point convolution, representing a feature aggregation of point cloud and object-level features. The pooled features are then concatenated and convolved to generate different attention coefficients, resulting in the output of the spatial attention network.

[0049] Finally, the output of the channel attention network is superimposed with the output of the spatial attention network, and the superimposed results are calculated point by point through a fully connected layer to obtain the attention score of each point cloud feature.

[0050] In the above example, through the channel and spatial attention modules, feature updates are performed point cloud by point cloud from a global perspective, and the interactions between complex points are fully learned, which helps to improve the accuracy of subsequent point cloud semantic segmentation.

[0051] For example, Figure 4 Figure 1 shows a network consisting of a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module. The kernel point convolution layer takes as input the spatial coordinates of a 3D point cloud. During training, the graph edge convolution layer takes as input the annotations of the 3D point cloud. During application, its input is also a 3D point cloud. The channel and spatial attention module calculates attention scores for the convolved features, further mining and extracting the global features of the point cloud. Its output is a point cloud feature matrix.

[0052] For example, Figure 5 As shown, the first network structure can use U-Net as the overall framework structure and use the five network layers in the virtual box for encoding. The first layer in the virtual box is used to perform vector conversion on the input data to obtain a vector representation of the input data, and the other five network layers are encoding layers. Each encoding layer includes multiple network layers. The network layer outside the virtual box is a decoding layer. Each network layer in the decoding layer can be composed of at least two of the graph edge convolution layer, the kernel point convolution layer, and the channel and spatial attention module. The network layer composed of these three can be as follows Figure 4 shown.

[0053] like Figure 4 As shown in the figure, during the training phase, the input data first passes through the kernel point convolutional encoding layer and the image edge conditional convolutional layer to extract local features and object-level features of the point cloud. Then, the features of the two are concatenated and input into the channel and spatial attention module. The channel and spatial attention module captures the input features at a deep level of global features, ultimately extracting local and global features of the point cloud.

[0054] like Figure 5 As shown in the figure, in order to capture geometric information at multiple scales, downsampling is used to gradually expand the receptive field of convolution. In the decoder, nearest neighbor upsampling is used to obtain the final point-by-point features. The model extracts local and global features in all encoding layers, and adds channel and spatial attention modules to the first, third, and fifth encoding layers to consider contextual information. Similarly, skip connections are used to pass the intermediate features of each encoding layer in the encoder to each decoding layer in the decoder. In the decoder, the intermediate features are connected with the upsampled features and then passed to Figure 5 The last network layer in , the fully connected layer, implements semantic prediction.

[0055] For example, the training data set, that is, the high-voltage transmission line point cloud data training set according to an embodiment of the present invention, can be obtained through collection and preprocessing in the following manner.

[0056] First, drones or airplanes are used to collect lidar data of high-voltage transmission lines in the first power grid, and the data is solved to obtain three-dimensional point cloud data. Then, the three-dimensional point cloud data is cropped and binned. For example, based on the route trajectory, the three-dimensional point cloud data is cropped along the transmission line and divided according to a fixed distance. Typically, the three-dimensional point cloud data between two or three towers is divided into a point cloud dataset. Next, the cropped and binned point cloud is denoised. For example, outliers are removed. At the same time, grid downsampling is used to downsample the denoised point cloud data, and a KD-Tree is constructed to facilitate the organization and indexing of the point cloud data.

[0057] Among them, the three-dimensional KD-Tree, abbreviated as k-dimensional tree, is a data structure for space partitioning. Specifically, KD-Tree is a binary tree structure for organizing k-dimensional space point data. Each non-leaf node can be divided into two subspaces by a hyperplane, and each corresponding subspace can be recursively divided in the same way. All subspaces are divided into left and right parts or into upper and lower parts. The partitioning of KD-Tree is performed along the coordinate axis, and all hyperplanes are perpendicular to the corresponding coordinate axis. KD-Tree is a relatively effective k-dimensional space point data organization structure, and has its own unique advantages in the field of high-dimensional space search (such as k-neighbor search). The use of three-dimensional KD-Tree here can efficiently organize and manage the initial point cloud data with a large amount of data.

[0058] The downsampling is grid sampling. Specifically, the 3D point cloud data is evenly divided into multiple small cubes, and the points within each cube are sampled. The mean of the points within each cube is used as the sampled point cloud for that cube. The number of each point cloud category within the cube is counted, and the point cloud category with the largest number of points is used as the sampled point cloud category. In this example, using grid sampling can greatly reduce the number of sampling points, reducing data calculation and memory consumption for subsequent model training and testing.

[0059] Finally, the sampled point cloud data are annotated to obtain the label information of each point cloud data, thereby constructing a high-voltage transmission line training dataset.

[0060] In one embodiment, the encoder includes 5 coding layers. From the input to the output of the encoder, the first coding layer, the third coding layer and the fifth coding layer in the encoder all adopt the structure of the second coding layer, and the second coding layer and the fourth coding layer in the encoder all adopt the architecture of the first coding layer.

[0061] In one embodiment, the decoder includes 5 decoding layers. From the input to the output of the decoder, the first decoding layer, the third decoding layer, and the fifth decoding layer in the decoder all adopt the structure of the second decoding layer, and the second decoding layer and the fourth decoding layer in the decoder all adopt the structure of the first decoding layer.

[0062] For example, Figure 5 As shown in , both the encoder and decoder are configured with 5 network layers. Figure 5 As shown in the figure, from left to right, the first, third, and fifth coding layers of the encoder all use the second coding layer, and the second and third coding layers of the encoder all use the first coding layer. The first, third, and fifth decoding layers of the decoder all use the second decoding layer, and the second and fourth decoding layers all use the first decoding layer.

[0063] It can be understood that the fifth encoding layer can be used as the first decoding layer of the decoder.

[0064] It can be understood that each encoding layer gradually downsamples the input features and gradually converts the dimensions of the features from 64 dimensions, 128 dimensions, 256 dimensions, 512 dimensions to 1024 dimensions.

[0065] It can be understood that each decoding layer gradually upsamples the input features, and gradually converts the dimensions of the features from 1024 dimensions, 512 dimensions, 256 dimensions, 128 dimensions to 64 dimensions.

[0066] In this way, a channel and spatial attention module is set every other network layer, and each network layer is provided with a parallel kernel point convolution layer and a graph edge convolution layer.

[0067] For example, in the direction from the input to the output of the encoder, the second coding layer and the fourth coding layer of the encoder both use the second coding layer, and the first coding layer, the third coding layer, and the fifth coding layer of the encoder all use the first coding layer. In the direction from the input to the output of the decoder, the first decoding layer, the third decoding layer, and the fifth decoding layer of the decoder all use the first decoding layer, and the second and fourth decoding layers both use the second decoding layer.

[0068] According to the above implementation, the point cloud segmentation model uses a network consisting of an encoder, a decoder, and a fully connected layer. Each network layer is equipped with a parallel kernel point convolution layer and a graph edge convolution layer to extract features of semantic orientation and inter-category geometric attributes, respectively. Channel and spatial attention modules are set in some network layers to mine contextual information within the scene and model local and global features. This network structure can improve the accuracy of semantic segmentation and semantic understanding capabilities.

[0069] In one embodiment, a first network structure is trained based on a training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model, including: training the first network structure based on the training set of high-voltage transmission line point cloud data to obtain a first semantic segmentation model; adjusting the structural type of each coding layer and the structural type of each decoding layer in the first network structure based on the network parameters of the first semantic segmentation model to obtain a second network structure, wherein the structural type of the coding layer includes a first coding layer and a second coding layer, and the structural type of the decoding layer includes a first decoding layer and a second decoding layer; adjusting the network parameters of the first semantic segmentation model based on the structural differences between the first network structure and the second network structure; setting the network parameters of the second network structure based on the adjusted network parameters of the first semantic segmentation model; training the second network structure after the network parameters have been set based on the training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model.

[0070] Exemplarily, the network parameters obtained by pre-training are used to determine the accuracy of the first network structure or the residual of each coding layer and its corresponding decoding layer, and these parameters are used to adjust the structural type of the coding layer and / or decoding layer to obtain the second network structure.

[0071] Exemplarily, the particle swarm algorithm may be used to adjust the structural type of each encoding layer and the structural type of each decoding layer in the first network structure to obtain the second network structure.

[0072] Illustratively, the training process for the first network structure and the training process for the second network structure may be the same or different.

[0073] Exemplarily, based on the network parameters of the first semantic segmentation model, the network parameters of the second network structure are set, and then based on the high-voltage transmission line point cloud data training set, the second network structure with the set network parameters is trained to obtain the target semantic segmentation model. This can reduce the number of training times or training time of the second network structure and maintain the accuracy of the obtained target semantic segmentation model.

[0074] Alternatively, the network parameters of the first semantic segmentation model are adjusted based on the structural differences between the first network structure and the second network structure as in the above example; the network parameters of the second network structure are set based on the adjusted network parameters of the first semantic segmentation model; and finally, the second network structure with the set network parameters is trained based on the high-voltage transmission line point cloud data training set to obtain the target semantic segmentation model. For example, if the structures of the two corresponding network layers in the first network structure and the second network structure are the same, the network parameters of the corresponding network layer in the first semantic segmentation model are not adjusted. If the structures of the two corresponding network layers in the first network structure and the second network structure are different, the network parameters of the corresponding network layer in the first semantic segmentation model are adjusted.

[0075] For example, during model training, the Adam optimizer can be used. Another example is using cross entropy as the loss function. Another example is using a warm-up strategy to gradually change the learning rate.

[0076] According to the above implementation, a semantic segmentation model is initially trained to obtain its network parameters. The network structure is then rapidly optimized by adjusting the structural types of the encoding and decoding layers within the network using these pre-trained network parameters. Finally, model training using this optimized network structure can reduce the number of training cycles or training time while maintaining or improving the accuracy of the target semantic segmentation model.

[0077] In one embodiment, based on the network parameters of the first semantic segmentation model, the structural type of each coding layer and the structural type of each decoding layer in the first network structure are adjusted to obtain a second network structure, including: based on the first network structure, determining an initial population, wherein the initial population includes multiple particles, each particle corresponds to a first network structure, and the first network structures corresponding to each particle have at least one coding layer or decoding layer with a different structural type; starting from the initial population, performing the following iterative operations: based on the network parameters of the first semantic segmentation model, setting the network parameters of the first network structure corresponding to each particle in this iterative population, to obtain a second semantic segmentation model corresponding to each particle in this iterative population; based on the high-voltage transmission line point cloud data training set, determining the semantic segmentation error of the second semantic segmentation model corresponding to each particle in this iterative population; based on the semantic segmentation error of the second semantic segmentation model corresponding to each particle in this iterative population, updating the individual optimal fitness value; when the individual optimal fitness value meets the preset fitness value condition, determining the second network structure based on the first network structure corresponding to the particle corresponding to the individual optimal fitness value.

[0078] In one embodiment, the above-mentioned iterative operation further includes: when the individual optimal fitness value does not meet the fitness value condition, adjusting the structure type of the encoding layer or decoding layer in the first network structure corresponding to each particle in the current iterative population to obtain the next iterative population.

[0079] Exemplarily, the network parameters of the first semantic segmentation model are used to set the network parameters of the first network structure corresponding to each particle to obtain the second semantic segmentation model corresponding to each particle, and then the semantic segmentation error of the second semantic segmentation model corresponding to each particle is calculated using the high-voltage transmission line point cloud data training set.

[0080] For example, before the first iteration, an initial value of the individual optimal fitness value is pre-set. In each iteration, if the minimum prediction error among the prediction errors of the second semantic segmentation model corresponding to each particle in the current iteration population is less than or equal to the individual optimal fitness value, the individual optimal fitness value is updated with the minimum prediction error. If the minimum prediction error is greater than the individual optimal fitness value, the individual optimal fitness value remains unchanged.

[0081] Exemplarily, the fitness value condition may be that the individual optimal fitness value is less than a preset fitness value threshold.

[0082] For example, if the number of iterations exceeds a preset threshold, a second network structure may be determined based on the first network structure corresponding to the particle corresponding to the individual optimal fitness value. If the number of iterations does not exceed the preset threshold, the structure type of the encoding layer or decoding layer in the first network structure corresponding to each particle in the current iterative population is adjusted to obtain the next iterative population.

[0083] According to the above embodiment, based on the first network structure, the population is initialized, and then the network parameters of the first semantic segmentation model are used starting from the initial population to optimize and iterate the particles in the population to obtain the final population and individual optimal fitness values, and the network structure corresponding to the individual optimal fitness values ​​is used to determine the final network structure. In this way, the network structure can be quickly optimized.

[0084] In one embodiment, the structural type of the coding layer or decoding layer in the first network structure corresponding to each particle in the current iterative population is adjusted to obtain the next iterative population, including: for each particle in the current iterative population, the following operations can be performed to obtain an iterative population: based on the residual between two adjacent coding layers in the first network structure corresponding to the particle and the respective corresponding decoding layers, the structural types of the two adjacent coding layers are adjusted; based on the adjusted structural types of the two adjacent coding layers, the structural types of the corresponding two decoding layers are adjusted to obtain the adjusted first network structure corresponding to the particle.

[0085] Exemplarily, if the residuals between two adjacent coding layers and their respective corresponding decoding layers are both greater than a preset residual threshold, the network structure of the coding layer that is ordered later among the two adjacent coding layers is set to the aforementioned second coding layer, and the network structure of the decoding layer corresponding to the coding layer that is ordered later is set to the aforementioned second decoding layer. The coding layer that is ordered earlier can use the first coding layer or the second coding layer, and its corresponding decoding layer can use the first decoding layer or the second decoding layer.

[0086] According to the above embodiment, the residual is used to determine the optimization method of the first network structure corresponding to each particle in the population, thereby improving the accuracy of population iteration and the speed of network structure optimization.

[0087] Figure 6 This is a structural block diagram of a semantic segmentation device for a high-voltage transmission line lidar point cloud according to an embodiment of the present invention.

[0088] like Figure 6 As shown, the semantic segmentation device may include:

[0089] A network structure determination module 610 is configured to construct a first network structure based on a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module;

[0090] A model training module 620 is configured to train the first network structure based on a training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model;

[0091] A semantic segmentation module 630 is configured to perform semantic segmentation on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model in response to the semantic segmentation request;

[0092] Wherein, the first network structure includes an encoder and a decoder;

[0093] The encoder includes at least one first encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel, and at least one second encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer connected in parallel and then connected in series with the channel and spatial attention module;

[0094] The decoder includes at least one first decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module.

[0095] In one embodiment, the encoder includes 5 coding layers. From the input to the output direction, the first coding layer, the third coding layer and the fifth coding layer in the encoder all adopt the structure of the second coding layer, and the second coding layer and the fourth coding layer in the encoder both adopt the architecture of the first coding layer.

[0096] In one embodiment, the decoder includes 5 decoding layers. From the input to the output direction, the first decoding layer, the third decoding layer and the fifth decoding layer in the decoder all adopt the structure of the second decoding layer, and the second decoding layer and the fourth decoding layer in the decoder all adopt the structure of the first decoding layer.

[0097] In one embodiment, the model training module 620 includes:

[0098] A first training unit is configured to train the first network structure based on the high-voltage transmission line point cloud data training set to obtain a first semantic segmentation model;

[0099] a structure adjustment unit, configured to adjust, based on the network parameters of the first semantic segmentation model, the structure type of each encoding layer and the structure type of each decoding layer in the first network structure to obtain a second network structure, wherein the structure type of the encoding layer includes the first encoding layer and the second encoding layer, and the structure type of the decoding layer includes the first decoding layer and the second decoding layer;

[0100] a network parameter adjustment unit, configured to adjust network parameters of the first semantic segmentation model based on structural differences between the first network structure and the second network structure;

[0101] a network parameter setting unit, configured to set network parameters of the second network structure based on the adjusted network parameters of the first semantic segmentation model;

[0102] The second training unit is used to train the second network structure after the network parameters have been set based on the high-voltage transmission line point cloud data training set to obtain the target semantic segmentation model.

[0103] In one embodiment, the structure adjustment unit is specifically used to:

[0104] Determining an initial population based on the first network structure, wherein the initial population includes a plurality of particles, each particle corresponds to a first network structure, and the first network structures corresponding to the particles have at least one encoding layer or decoding layer with a different structure type;

[0105] Starting from the initial population, the following iterative operations are performed:

[0106] Based on the network parameters of the first semantic segmentation model, the network parameters of the first network structure corresponding to each particle in the current iteration population are set respectively to obtain the second semantic segmentation model corresponding to each particle in the current iteration population;

[0107] Determining, based on the high-voltage transmission line point cloud data training set, a semantic segmentation error of the second semantic segmentation model corresponding to each particle in the current iterative population;

[0108] Based on the semantic segmentation error of the second semantic segmentation model corresponding to each particle in the current iterative population, updating the individual optimal fitness value;

[0109] When the individual optimal fitness value meets a preset fitness value condition, the second network structure is determined based on the first network structure corresponding to the particle corresponding to the individual optimal fitness value.

[0110] In one embodiment, the iterative operation further includes:

[0111] When the individual optimal fitness value does not meet the fitness value condition, the structure type of the encoding layer or the decoding layer in the first network structure corresponding to each particle in the current iterative population is adjusted to obtain the next iterative population.

[0112] In one embodiment, adjusting the structure type of the encoding layer or the decoding layer in the first network structure corresponding to each particle in the current iteration population includes:

[0113] Adjusting the structure types of the two adjacent coding layers based on the residuals between the two adjacent coding layers and the respective corresponding decoding layers in the first network structure corresponding to the particle;

[0114] Based on the adjusted structural types of the two adjacent encoding layers, the structural types of the corresponding two decoding layers are adjusted to obtain the adjusted first network structure corresponding to the particle.

[0115] For the description of specific functions and examples of each module and submodule of the system in the embodiment of the present invention, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0116] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0117] According to an embodiment of the present invention, the present invention further provides a system and a readable storage medium.

[0118] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0119] like Figure 7 As shown, electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of electronic device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0120] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0121] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the semantic segmentation method for high-voltage transmission line LiDAR point clouds. For example, in some embodiments, the semantic segmentation method for high-voltage transmission line LiDAR point clouds can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the semantic segmentation method for high-voltage transmission line LiDAR point clouds described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the semantic segmentation method of the high-voltage transmission line lidar point cloud in any other appropriate manner (for example, by means of firmware).

[0122] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0128] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0129] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A semantic segmentation method for high-voltage transmission line lidar point cloud, characterized in that: include: Based on the kernel point convolution layer, the graph edge convolution layer, and the channel and spatial attention module, a first network structure is constructed, wherein the first network structure includes an encoder and a decoder; the encoder includes at least one first encoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second encoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module; the decoder includes at least one first decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second decoding layer composed of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module; Based on a training set of high-voltage transmission line point cloud data, the first network structure is trained to obtain a target semantic segmentation model, including: when performing an iterative operation, if the residuals between two adjacent coding layers and their respective corresponding decoding layers in the first network structure are both greater than a preset residual threshold, then setting the network structure of the coding layer that is sorted later among the two adjacent coding layers as the second coding layer, and setting the network structure of the decoding layer corresponding to the coding layer that is sorted later as the second decoding layer, wherein the coding layer that is sorted earlier is set as the first coding layer, and the decoding layer corresponding to the coding layer that is sorted earlier is set as the first decoding layer; In response to the semantic segmentation request, semantic segmentation is performed on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model.

2. The method according to claim 1, characterized in that The encoder includes 5 coding layers. From the input to the output direction, the first coding layer, the third coding layer and the fifth coding layer in the encoder all adopt the structure of the second coding layer, and the second coding layer and the fourth coding layer in the encoder both adopt the architecture of the first coding layer.

3. The method according to claim 1, characterized in that The decoder includes 5 decoding layers. From input to output, the first decoding layer, the third decoding layer and the fifth decoding layer in the decoder all adopt the structure of the second decoding layer, and the second decoding layer and the fourth decoding layer in the decoder all adopt the structure of the first decoding layer.

4. The method according to claim 1, wherein The first network structure is trained based on the high-voltage transmission line point cloud data training set to obtain a target semantic segmentation model, including: Based on the high-voltage transmission line point cloud data training set, the first network structure is trained to obtain a first semantic segmentation model; Based on the network parameters of the first semantic segmentation model, adjusting the structure type of each encoding layer and the structure type of each decoding layer in the first network structure to obtain a second network structure, wherein the structure type of the encoding layer includes the first encoding layer and the second encoding layer, and the structure type of the decoding layer includes the first decoding layer and the second decoding layer; Adjusting network parameters of the first semantic segmentation model based on structural differences between the first network structure and the second network structure; Setting network parameters of the second network structure based on the adjusted network parameters of the first semantic segmentation model; Based on the high-voltage transmission line point cloud data training set, the second network structure after the network parameters have been set is trained to obtain the target semantic segmentation model.

5. The method according to claim 4, characterized in that The network parameters based on the first semantic segmentation model are used to adjust the structural types of the encoding layers and the structural types of the decoding layers in the first network structure to obtain a second network structure, including: Determining an initial population based on the first network structure, wherein the initial population includes a plurality of particles, each particle corresponds to a first network structure, and the first network structures corresponding to the particles have at least one encoding layer or decoding layer with a different structure type; Starting from the initial population, the following iterative operations are performed: Based on the network parameters of the first semantic segmentation model, the network parameters of the first network structure corresponding to each particle in the current iteration population are set respectively to obtain the second semantic segmentation model corresponding to each particle in the current iteration population; Determining, based on the high-voltage transmission line point cloud data training set, a semantic segmentation error of the second semantic segmentation model corresponding to each particle in the current iterative population; Based on the semantic segmentation error of the second semantic segmentation model corresponding to each particle in the current iterative population, updating the individual optimal fitness value; When the individual optimal fitness value meets a preset fitness value condition, the second network structure is determined based on the first network structure corresponding to the particle corresponding to the individual optimal fitness value.

6. The method according to claim 5, characterized in that The iterative operation further includes: When the individual optimal fitness value does not meet the fitness value condition, the structure type of the encoding layer or the decoding layer in the first network structure corresponding to each particle in the current iterative population is adjusted to obtain the next iterative population.

7. A semantic segmentation device for high-voltage transmission line lidar point cloud, characterized in that: include: A network structure determination module, configured to construct a first network structure based on a kernel point convolution layer, a graph edge convolution layer, and a channel and spatial attention module, wherein the first network structure includes an encoder and a decoder; the encoder includes at least one first encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second encoding layer consisting of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module; the decoder includes at least one first decoding layer consisting of the kernel point convolution layer and the graph edge convolution layer in parallel, and at least one second decoding layer consisting of the kernel point convolution layer and the graph edge convolution layer in parallel and then connected in series with the channel and spatial attention module; A model training module, configured to train the first network structure based on a training set of high-voltage transmission line point cloud data to obtain a target semantic segmentation model, comprising: when performing an iterative operation, if the residuals between two adjacent coding layers and their respective corresponding decoding layers in the first network structure are both greater than a preset residual threshold, setting the network structure of the coding layer that is ordered later among the two adjacent coding layers to be the second coding layer, and setting the network structure of the decoding layer corresponding to the coding layer that is ordered later to be the second decoding layer, wherein the coding layer that is ordered earlier is set to be the first coding layer, and the decoding layer corresponding to the coding layer that is ordered earlier is set to be the first decoding layer; The semantic segmentation module is used to respond to the semantic segmentation request and perform semantic segmentation on the high-voltage transmission line point cloud data corresponding to the semantic segmentation request based on the target semantic segmentation model.

8. A semantic segmentation system for high-voltage transmission line lidar point clouds, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the processor, and the processor is used to obtain the instructions from the memory and execute the instructions, so that the processor can execute the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to be provided to a computer, so as to enable the computer to execute the method according to any one of claims 1 to 6.

Images