Ancient building point cloud semantic segmentation method and system based on lightweight neural network

Abstract

The application discloses a light neural network-based ancient building point cloud semantic segmentation method and system, which comprises the following steps: constructing a neural network with an encoder-decoder structure, compressing the neural network through a self-distillation module, and generating an initial light neural network, wherein the encoder uses a local feature aggregation module to extract feature information; obtaining an ancient building point cloud data sample set as the input of the initial light neural network, performing network training, introducing a contrast boundary learning module, using random sampling to down-sample the point cloud, and constructing a light neural network; based on the light neural network, taking the to-be-tested point cloud data as the network input, and obtaining a point cloud semantic segmentation result; the application can maintain high semantic segmentation accuracy, improve the efficiency of the semantic segmentation algorithm, reduce the requirements on equipment, and is an effective solution for large-scale ancient building point cloud semantic segmentation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ancient building point cloud data semantic segmentation, in particular to an ancient building point cloud semantic segmentation method and system based on a lightweight neural network. BACKGROUND

[0002] With the widespread attention to ancient buildings and cultural relics, the size and various architectural details of ancient buildings can be more realistically obtained by using three-dimensional laser scanning technology, and the automatic segmentation of various wooden components of ancient building point clouds by using point cloud semantic segmentation has become an important part of the later fine protection and repair of ancient buildings. Traditional ancient building semantic segmentation algorithms usually rely too much on artificial design features and are easily affected by noise and data, so the efficiency is low in large-scale scenarios. In recent years, researchers have turned their attention to using deep learning to automatically extract features. Since Charles et al. proposed the PointNet network for semantic segmentation in 2017, a large number of point cloud semantic segmentation methods have emerged, such as PointNet++, DGCNN, RandLA-Net, etc. And it has been proven that the semantic segmentation method based on deep learning has more advantages than the traditional semantic segmentation algorithm, and has been widely used in automatic segmentation of point cloud information.

[0003] However, due to the large amount of point cloud data, there are often problems of large amount of calculation, long training time, and high requirements for equipment. Therefore, it is urgent to design an ancient building point cloud semantic segmentation method based on a lightweight neural network, which can improve the segmentation accuracy while greatly improving the semantic segmentation efficiency of the point cloud and reducing the requirements for the equipment. SUMMARY

[0004] In order to solve the above problems, the purpose of the present application is to provide a method for solving the problem of low running efficiency of the segmentation algorithm and high requirements for the running equipment of the existing semantic segmentation network when facing large-scale point cloud data due to the parameter quantity and the amount of calculation.

[0005] In order to achieve the above technical purpose, the present application provides an ancient building point cloud semantic segmentation method based on a lightweight neural network, which comprises the following steps:

[0006] A neural network with an encoder-decoder structure is constructed, the neural network is compressed through a self-distillation module, and an initial lightweight neural network is generated, wherein the encoder uses a local feature aggregation module to extract feature information;

[0007] An ancient building point cloud data sample set is obtained as the input of the initial lightweight neural network, network training is performed, and a contrast boundary learning module is introduced to use random sampling to downsample the point cloud and construct a lightweight neural network;

[0008] Based on the lightweight neural network, the point cloud data to be measured is taken as the network input, and a point cloud semantic segmentation result is obtained.

[0009] Preferably, in the process of constructing the neural network, the encoder is composed of a self-attention convolution integration module and a local spatial encoding module, wherein the self-attention convolution integration module is used for weighted fusion of shallow features and deep features, and the point cloud is enriched and the features of the points are learned through the local feature encoding module of each layer.

[0010] Preferably, in the process of weighted fusion of shallow features and deep features through the self-attention convolution integration module, the input features are mapped by using convolution to obtain an intermediate feature set, and the intermediate features are reused and aggregated to realize fusion.

[0011] Preferably, in the process of reusing and aggregating the intermediate features, the intermediate features are reused and aggregated through the Self-Attention method and the convolution method.

[0012] Preferably, in the process of generating the initial lightweight neural network, the self-distillation module is used to rearrange the order sampling by limiting half of each small batch of samples to coincide with the previous iteration, while the other half will coincide with the upcoming iteration, and then the first half of the small batch instantaneously refines the soft target generated in the last iteration to realize the compression of the neural network.

[0013] Preferably, in the process of down-sampling the point cloud through the contrast boundary learning module, the features around the learning boundary region are obtained by probing on multiple scales, labeling the sub-scene at each sub-sampling, and iteratively determining the label using the sub-sampling program to enhance the cross-boundary feature recognition capability.

[0014] The application also discloses an ancient building point cloud semantic segmentation system based on a lightweight neural network, comprising:

[0015] The neural network compression module is used for constructing a neural network with an encoder-decoder structure, compressing the neural network through the self-distillation module, and generating an initial lightweight neural network, wherein the encoder extracts feature information by using the local feature aggregation module.

[0016] The semantic segmentation network construction module is used for taking an ancient building point cloud data sample set as the input of the initial lightweight neural network, training the network, and constructing a lightweight neural network by introducing the contrast boundary learning module and down-sampling the point cloud through random sampling.

[0017] The semantic segmentation module is used for taking the point cloud data to be measured as the network input based on the lightweight neural network, and obtaining a point cloud semantic segmentation result.

[0018] The present application discloses the following technical effects:

[0019] Compared with the prior art, the lightweight semantic segmentation network built by the self-distillation module reduces the loss of geometric features by using the self-attention convolution integration module, gradually increases the receptive field of each point, and thus improves the learning ability of the internal features of the point cloud, and adopts a sub-scene boundary mining strategy to focus on learning the edge area of the ground object, thereby improving the semantic segmentation accuracy. At the same time, the self-distillation idea is introduced, and the model is lightened by self-distillation in each batch. Therefore, the present application improves the efficiency of the semantic segmentation algorithm while maintaining high semantic segmentation accuracy, reduces the requirements for the device, and is an effective solution for large-scale ancient building point cloud semantic segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a method flowchart of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] As Figure 1 shown, the present application provides an ancient building point cloud semantic segmentation method based on a lightweight neural network, including the following steps:

[0024] A neural network with an encoder-decoder structure is constructed, the neural network is compressed by a self-distillation module, and an initial lightweight neural network is generated, wherein the encoder uses a local feature aggregation module to extract feature information;

[0025] The point cloud data sample set of the ancient building is acquired as the input of the initial lightweight neural network, network training is carried out, a lightweight neural network is constructed by introducing a contrast boundary learning module and using random sampling to down-sample the point cloud;

[0026] Based on the lightweight neural network, the point cloud semantic segmentation result is obtained by taking the to-be-tested point cloud data as the network input.

[0027] Further preferably, in the process of constructing the neural network, the encoder mentioned in the application is composed of a self-attention convolution integration module and a local spatial encoding module, wherein the self-attention convolution integration module is used for weighting and fusing shallow features and deep features, and the point cloud is enriched and learns the features of the points through the local feature encoding module of each layer.

[0028] Further preferably, in the process of weighting and fusing shallow features and deep features through the self-attention convolution integration module, the application uses convolution to map the input features to obtain an intermediate feature set, and then reuses and aggregates the intermediate features to realize fusion.

[0029] Further preferably, in the process of reusing and aggregating the intermediate features, the application reuses and aggregates the intermediate features through the Self-Attention method and the convolution method.

[0030] Further preferably, in the process of generating the initial lightweight neural network, the self-distillation module mentioned in the application is used to rearrange the order of sampling by limiting half of each small batch of samples to coincide with the previous iteration, while the other half will coincide with the upcoming iteration, and then the first half of the small batch instantaneously refines the soft target generated in the last iteration to realize the compression of the neural network.

[0031] Further preferably, in the process of down-sampling the point cloud through the contrast boundary learning module, the application probes on multiple scales, labels each sub-sampling, iteratively determines the label using a sub-sampling program, and then realizes the acquisition of features around the learning boundary area, and enhances the cross-boundary feature recognition capability.

[0032] The application also discloses an ancient building point cloud semantic segmentation system based on a lightweight neural network, comprising:

[0033] The neural network compression module is used for constructing a neural network with an encoder-decoder structure, compressing the neural network through the self-distillation module to generate an initial lightweight neural network, wherein the encoder uses a local feature aggregation module to extract feature information.

[0034] The semantic segmentation network construction module is used for acquiring the ancient building point cloud data sample set as the input of an initial lightweight neural network, performing network training, and constructing a lightweight neural network by introducing a contrast boundary learning module and using random sampling to downsample the point cloud.

[0035] The semantic segmentation module is used for obtaining a point cloud semantic segmentation result based on the lightweight neural network and taking the to-be-tested point cloud data as the network input.

[0036] The application further discloses a computer program for realizing the logical process of the ancient building point cloud semantic segmentation method, and further forms an executable program embedded in an intelligent device for collecting ancient building point cloud data and performing semantic segmentation.

[0037] The application further discloses a mobile storage device for carrying an ancient building point cloud semantic segmentation system and interacting with an intelligent device, and serving as an external device of the intelligent device to assist the intelligent device in performing semantic segmentation, so that the intelligent device has the function of semantic segmentation of ancient building point cloud data and the function and practicability of the device are improved.

[0038] Embodiment 1: The ancient building point cloud semantic segmentation method of the lightweight neural network disclosed in the application belongs to a network structure model, and solves the defects of the prior art that the model structure of the point cloud semantic segmentation method is complex and requires high-performance devices. The method comprises network construction, training and testing. The network has an encoding-decoding structure, and after network training based on a point cloud data sample set, a test set is input into the network to obtain a segmentation result of the point cloud data. On one hand, the method firstly reduces the calculation complexity and improves the key feature acquisition capability through attention convolution integration, and then enhances the discrimination capability of cross-boundary features through contrast boundary learning, so as to improve the efficiency and precision of large-scale point cloud semantic segmentation. On the other hand, self-knowledge refinement is performed through self-distillation, so that the proposed model can accurately and efficiently realize the semantic segmentation of point clouds.

[0039] The application provides an ancient building point cloud semantic segmentation method of a lightweight neural network, and specifically comprises the following processes: network construction, training and testing. The network is a deep semantic segmentation network with an encoder-decoder structure constructed by a pytorch deep learning framework, and after network training based on a point cloud data sample set, to-be-tested ancient building point cloud data is input into the network to obtain a point cloud semantic segmentation result. The network training comprises the following steps:

[0040] Step 1: The ancient building point cloud data sample is divided into a training set, a verification set and a test set.

[0041] Step 2: read in the three-dimensional point cloud data of the training set and the corresponding label point cloud, in order to make full use of the training sample set, the original large point cloud data and the label data are randomly sampled, the sampling frequency of each round of training is set, and the sampling size parameter is set according to the size of the display memory, and the point cloud data and the label data are randomly sampled at the same time.

[0042] Step 3: set the learning rate, exponential decay rate and regularization coefficient to train the depth network, and select the depth network with the highest verification set accuracy for semantic segmentation of ancient building point cloud data.

[0043] The lightweight neural network mentioned in the application has an encoder-decoder structure, and after network training based on the point cloud data sample set, the measured point cloud data is used as the network input to obtain the point cloud semantic segmentation result.

[0044] The encoder uses a local feature aggregation module to extract feature information, which includes a self-attention convolution integration module and a local spatial encoding module, the self-attention convolution integration module fuses the shallow features and the deep features by weighting, first, the input features are mapped by using convolution to obtain a rich intermediate feature set, then the intermediate features are reused and aggregated according to different modes (in Self-Attention mode and convolution mode respectively). The point cloud is enriched and the features of the learning points are learned through the local feature encoding module of each layer.

[0045] On this basis, the feature recognition ability across the border is enhanced, in order to enhance the acquisition ability of edge features, a contrast boundary learning module is adopted, random sampling is used to downsample the point cloud, in the sampling process, the boundary points in each subsampled point cloud are found, for the boundary points, it learns to be more similar to the adjacent points in the same category and more different from the other adjacent points in different categories, but since the label of the subsampled point is not defined, it is difficult to get the correct definition of the boundary point set, therefore, in order to better explore the boundary scene, the exploration is carried out at multiple scales, the sub-scene labeling is carried out at each subsampling, the sub-sampling program is used to iteratively determine the label, and finally the features around the boundary area are better learned.

[0046] After achieving good segmentation accuracy, the model is lightened. By introducing a self-distillation module for model compression, in the training stage, it rearranges the order sampling by limiting half of each mini-batch to coincide with the previous iteration. At the same time, the remaining half will coincide with the upcoming iteration. Then, the first half of the mini-batch refines the soft target generated in the last iteration. In this mechanism, the network plays a dual role of teacher and student in each mini-batch of training, as a teacher, it provides a soft target to regulate itself in the next iteration. As a student, it refines the smooth label generated in the last iteration and minimizes the supervised learning target, and finally achieves a good lightening effect.

[0047] For model lightening using self-distillation technology of self-distillation module, knowledge distillation can also be used instead, which transfers knowledge from powerful but cumbersome network to lightweight network for model compression. However, it also has disadvantages, either requiring additional network structure modification or being difficult to parallelize. In addition, network compression technology such as pruning can effectively remove network weights or neurons while maintaining accuracy. Since pruning simplifies the neural network, the knowledge provided by the pruned teacher model is more friendly to the student model and is easier to transfer. In addition, pruning can be performed before distillation, the complex model is pruned to improve the ability of knowledge extraction, and the knowledge is transferred from the sparse network (unstructured pruning) to the network with fewer channels for knowledge distillation to achieve the effect of model compression.

[0048] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowchart and / or block diagram. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0049] In the description of the present application, it should be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0050] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for ancient building point cloud semantic segmentation based on a lightweight neural network, characterized in that, The method comprises the following steps: constructing a neural network with an encoder-decoder structure, compressing the neural network through a self-distillation module, and generating an initial lightweight neural network, wherein the encoder extracts feature information by using a local feature aggregation module; obtaining an ancient building point cloud data sample set as an input of the initial lightweight neural network, performing network training, and constructing a lightweight neural network by introducing a contrast boundary learning module and using random sampling to down-sample the point cloud; based on the lightweight neural network, obtaining a point cloud semantic segmentation result by taking to-be-tested point cloud data as network input; in the process of constructing the neural network, the encoder is composed of a self-attention convolution integration module and a local spatial encoding module, wherein the self-attention convolution integration module is used for weighted fusion of shallow features and deep features, and the point cloud is enriched and learns the features of the points through the local feature encoding module of each layer; in the process of generating the initial lightweight neural network, the self-distillation module is used to rearrange the order sampling by limiting half of each small batch sample to coincide with the previous iteration, while the other half will coincide with the upcoming iteration, then the first half of the small batch instantaneously refines the soft target generated in the last iteration, and the compression of the neural network is realized; in the process of down-sampling the point cloud through the contrast boundary learning module, the features around the boundary region are obtained by probing on multiple scales, labeling the sub-scene at each sub-sampling, and iteratively determining the label by using a sub-sampling program, so as to enhance the cross-boundary feature recognition capability.

2. The ancient building point cloud semantic segmentation method based on the lightweight neural network according to claim 1, wherein: in the process of weighted fusion of shallow features and deep features through the self-attention convolution integration module, the intermediate feature set is obtained by using convolution to map the input features, and the intermediate features are reused and aggregated, so as to realize fusion.

3. The ancient building point cloud semantic segmentation method based on the lightweight neural network according to claim 2, wherein: in the process of reusing and aggregating the intermediate features, the intermediate features are reused and aggregated through the Self-Attention method and the convolution method.

4. The ancient building point cloud semantic segmentation method based on the lightweight neural network according to claim 1, wherein: in the process of compressing the neural network, a self-distillation network compression method is adopted, and the self-distillation module compresses the neural network after self-iteration refinement in each batch, wherein the self-distillation module has a self-distillation function or a knowledge distillation function.

5. A system for lightweight neural network based point cloud semantic segmentation of ancient buildings, configured to perform the lightweight neural network based point cloud semantic segmentation method of ancient buildings according to claim 1, characterized in that, comprises: a neural network compression module, configured to construct a neural network with an encoder-decoder structure, compress the neural network through a self-distillation module, and generate an initial lightweight neural network, wherein the encoder extracts feature information by using a local feature aggregation module; The semantic segmentation network construction module is configured to acquire an ancient building point cloud data sample set as an input of the initial lightweight neural network, perform network training, and construct a lightweight neural network by introducing a contrast boundary learning module and using random sampling to down-sample the point cloud; The semantic segmentation module is configured to obtain a point cloud semantic segmentation result based on the lightweight neural network and using to-be-tested point cloud data as a network input.

Images