Point cloud-oriented end-to-end instance-level three-dimensional change detection method and system

Abstract

The invention discloses a point cloud-oriented end-to-end instance-level three-dimensional change detection method and system, and the method comprises the steps: obtaining the three-dimensional point cloud data of a target object in a to-be-detected region, and arranging the three-dimensional point cloud data into dual-time-sequence point cloud data; inputting the double-time-sequence point cloud data into the trained end-to-end three-dimensional instance change detection model to obtain a three-dimensional instance change detection result; the end-to-end three-dimensional instance change detection model is constructed based on deep learning, and comprises a multi-task shared feature encoder for outputting point cloud coding basic features of dual-time sequence; the feature processing double branches comprise a change feature optimized three-dimensional instance segmentation branch and an instance feature guided three-dimensional change detection branch, and a three-dimensional change detection result and a three-dimensional instance segmentation result are respectively output based on the point cloud coding basic features; and the information fusion module is used for fusing the three-dimensional change detection result and the three-dimensional instance segmentation result to obtain a three-dimensional instance change detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional GIS, and in particular to a point cloud-oriented end-to-end instance-level three-dimensional change detection method and system. BACKGROUND

[0002] A large number of urban renewal tasks not only require monitoring whether changes have occurred and which type of ground object has changed, but also require monitoring the fine-grained changes of each ground object instance, such as the area and height of a single building. In order to solve this problem, instance change detection technology needs to be used. Two-dimensional instance change detection is limited by the characteristics of image data, and can only detect changes in the planar dimension of instances, lacking the ability to detect changes in three-dimensional space. Therefore, three-dimensional instance change detection is needed.

[0003] The mainstream three-dimensional instance change detection product is three-dimensional point cloud. At present, there are few related methods for three-dimensional instance change detection, and almost all of them implement three-dimensional instance change detection based on the idea of multiple stages. The multi-stage method decomposes the three-dimensional instance change detection into three-dimensional change detection and three-dimensional instance segmentation tasks, and uses different special models for different tasks to solve the task requirements. However, the architecture of multiple models linear combination not only increases the complexity of the model and the amount of calculation, but also because of the characteristics of the serial structure, it is difficult to optimize the overall performance of the model, resulting in poor comprehensive three-dimensional instance change detection performance of the model.

[0004] In summary, in order to realize the fine-grained management of ground object instances in the city, three-dimensional instance change detection is needed. However, the existing three-dimensional instance change detection technology still has the following problems: (1) the current mainstream multi-task multi-model architecture has high algorithm complexity, and due to the characteristics of multiple model linear combination, the connection between models is weak, and the comprehensive performance of the model is difficult to optimize. (2) Three-dimensional instance segmentation and three-dimensional change detection are two strongly related tasks, and simple linear combination cannot realize the performance coupling of the model structure, making it difficult to exert the performance of each task. (3) There is currently a lack of end-to-end three-dimensional instance change detection technology, and the precision and efficiency of the three-dimensional instance change detection task need to be improved. Therefore, it is urgent to research a multi-task deep-coupled end-to-end three-dimensional instance change detection method. SUMMARY

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a point cloud-oriented end-to-end three-dimensional instance change detection method and system based on deep learning, which effectively couples multiple task modules in a single model by proposing an end-to-end three-dimensional instance change detection model based on deep learning, effectively completes the mutual complementation and optimization of three-dimensional change detection and three-dimensional instance segmentation, obtains more rich data representation, and realizes efficient city-level three-dimensional instance change detection which can effectively identify changes existing in each ground object instance and identify changes in each instance while segmenting the instance.

[0006] According to an aspect of the present application, a point cloud-oriented end-to-end three-dimensional instance change detection method based on deep learning is provided, comprising:

[0007] The training of the end-to-end three-dimensional instance change detection model comprises:

[0008] The end-to-end three-dimensional instance change detection model based on deep learning comprises a multi-task shared feature encoder for extracting encoded basic features of input double-time point cloud data, a feature processing double-branch including a three-dimensional instance segmentation branch optimized for change features for processing three-dimensional instance segmentation results obtained from the encoded basic features, and a three-dimensional change detection branch guided by instance features for processing three-dimensional change detection results output from the encoded basic features, and an information fusion module for fusing the three-dimensional change detection results and the three-dimensional instance segmentation results to obtain three-dimensional instance change detection results.

[0009] The training data set is constructed for model training, and the trained end-to-end three-dimensional instance change detection model is output.

[0010] As a further embodiment, the multi-task shared feature encoder is a twin network structure, and the single network structure is a multi-level down-sampling structure comprising a plurality of feature encoding modules connected in sequence, which inputs single-time point cloud data and outputs encoded basic features of single-time point cloud data at different resolutions. The twin network structure processes single-time point cloud data at different times in the double-time point cloud data respectively, and outputs encoded basic features of the double-time point cloud data.

[0011] As a further implementation, the feature encoding module comprises a sampling layer, a feature extraction layer, and a pooling layer; the sampling layer uses a grid downsampling method to perform feature point sampling operations on the input to generate a plurality of feature kernel points; the feature extraction layer comprises a plurality of layers of feature extraction structures, a single layer structure includes a KPConv module, a batch normalization operation, and a ReLU activation operation, for the feature kernel points of a plurality of sampling layers, a KNN algorithm is used to construct a local neighborhood, in the KPConv module, the features are weighted extracted according to the spatial distribution of the points in the local field, and are connected with the input features in residual, the original geometric information is retained, and finally the batch normalization and ReLU activation operations are performed to obtain the feature extraction result of the layer; the pooling layer processes the feature extraction result of the last layer of feature extraction structure, aggregates high-level feature information using a cross-step KPConv downsampling, and outputs the encoded basic features.

[0012] As a further implementation, the three-dimensional instance segmentation branch includes an instance refinement module and an instance segmentation decoder, the point cloud encoded basic features are input into the instance segmentation decoder for step-by-step processing to obtain instance-specific features, the instance refinement module obtains the mask information and the category information of the instance based on the instance-specific features, the initial point cloud encoded basic features, and the query to obtain the three-dimensional instance segmentation result.

[0013] As a further implementation, the instance segmentation decoder is composed of a plurality of layers of instance decoding modules connected in sequence, the number of layers is one less than the number of layers of the feature encoder network structure, the input point cloud encoded basic features are input into the first layer of instance decoding module, and the layers are processed in sequence, and the last layer of instance decoding module outputs the instance-specific features, wherein each layer of instance decoding module is composed of a convolution layer and an up-sampling layer, the convolution layer uses CNN convolution operation to transform the feature space, and the up-sampling layer uses the nearest neighbor point up-sampling method to restore the point cloud features.

[0014] As a further implementation, the instance refinement module is composed of an instance mask calculation module and an instance query refiner module, the initial query and the instance-specific features are input into the instance mask calculation module, the similarity scores between points are obtained as attention masks through cross-multiplication, and the filtered results are used as the mask information of the instance according to the preset attention mask threshold; the attention mask, the instance-specific features, and the query are input into the instance query refiner module, the query is optimized using the Transformer structure, and the category information of the instance is output.

[0015] As a further implementation, the three-dimensional change detection branch is composed of a plurality of layers of change information extraction modules and a plurality of layers of change feature decoders, the number of layers of the change information extraction module is consistent with the number of layers of the feature encoder network structure, and the number of layers of the change feature decoder is one more than the number of layers of the feature encoder network structure.

[0016] The change information extraction module is configured to extract the change-specific feature of the encoded base feature, extract and output the change-specific feature of the current layer, and match the neighborhood relationship of the double-time point cloud by using the KNN algorithm in the change information extraction module, and then extract the feature difference by calculating the feature difference value to obtain the change-specific feature.

[0017] The first change feature decoder takes the double-time point cloud encoded base feature and the change-specific feature output by the first change information extraction module as input, and outputs the decoding result, and the input of the subsequent change feature decoder is the change-specific feature output by the change information extraction module of the corresponding layer and the decoding result of the change feature decoder of the previous layer, which is processed level by level, and the output of the last change feature decoder is spliced and fused with the output of the last change feature decoder to obtain the final three-dimensional change detection result.

[0018] As a further embodiment, the feature fusion module detects the changed target instance point cloud according to the three-dimensional change detection result and the three-dimensional instance segmentation result, and outputs the three-dimensional change detection result and the three-dimensional instance segmentation result of the target instance point cloud as the three-dimensional instance change detection result.

[0019] As a further embodiment, the loss value in the training process of the end-to-end three-dimensional instance change detection model is a comprehensive loss value, wherein the comprehensive loss value is the weighted sum of the three-dimensional instance segmentation result loss value and the three-dimensional change detection result loss value.

[0020] According to another aspect of the present specification, an end-to-end three-dimensional instance change detection system based on deep learning for point cloud is provided, comprising:

[0021] The data acquisition and arrangement module acquires the three-dimensional point cloud data of the target object in the target area and arranges it into double-time point cloud data;

[0022] The three-dimensional instance change detection module processes the double-time data and outputs the three-dimensional instance change detection result;

[0023] The training of the end-to-end three-dimensional instance change detection model comprises:

[0024] The end-to-end three-dimensional instance change detection model based on deep learning is constructed, including: a multi-task shared feature encoder used for extracting encoded basic features of input double time point cloud data; a feature processing double branch including a three-dimensional instance segmentation branch optimized by change features, used for processing three-dimensional instance segmentation results obtained by the encoded basic features, and a three-dimensional change detection branch guided by instance features, used for processing three-dimensional change detection results output by the encoded basic features; and an information fusion module used for fusing the three-dimensional change detection results and the three-dimensional instance segmentation results to obtain three-dimensional instance change detection results.

[0025] The training data set is constructed to train the model, and the trained end-to-end three-dimensional instance change detection model is output.

[0026] Compared with the prior art, the beneficial effects of the present application are that the present application proposes an end-to-end three-dimensional instance change detection model based on deep learning, which effectively couples multiple task modules in a single model, effectively completes the mutual complementation and optimization of three-dimensional change detection and three-dimensional instance segmentation, obtains more rich data representation, and realizes efficient city-level three-dimensional instance change detection, which can effectively identify changes existing in each ground object instance, and identify changes in each instance while segmenting the instance. BRIEF DESCRIPTION OF DRAWINGS

[0027] To make the technical solutions of the present application or prior art clearer, the drawings used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is a flowchart of a point cloud-oriented end-to-end instance-level three-dimensional change detection method in the embodiment of the present application;

[0029] Figure 2 is a general structure diagram of all neural network models in the embodiment of the present application;

[0030] Figure 3 is a structure diagram of the multi-task shared feature encoder in the embodiment of the present application;

[0031] Figure 4 is a structure diagram of the three-dimensional instance segmentation branch optimized by change features in the embodiment of the present application;

[0032] Figure 5 is a structure diagram of the three-dimensional change detection branch guided by instance features in the embodiment of the present application;

[0033] Figure 6 is an effect diagram of three-dimensional instance change detection in the embodiment of the present application;

[0034] Figure 7 is a structural schematic diagram of a point cloud-oriented end-to-end instance-level three-dimensional change detection system in an embodiment of the present application. DETAILED DESCRIPTION

[0035] It should be noted that:

[0036] The terms "comprising" and "having" and any variations thereof in the specification and claims and the above drawings, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units, not necessarily limited to which steps or units are clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application are combined with each other at will to form new technical solutions, and such combination is not restricted by the order of steps and / or structure composition mode, but must be based on the realization by those skilled in the art, when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that such combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0038] As shown in Figure 1 , is a flowchart of a point cloud-oriented end-to-end three-dimensional instance change detection method based on deep learning in an embodiment of the present application, the method comprising: Figure 1

[0039] Obtaining the target three-dimensional point cloud data of the area to be measured, and arranging it into double-time-series point cloud data;

[0040] Inputting the double-time-series point cloud data into the trained end-to-end three-dimensional instance change detection model to obtain the three-dimensional instance change detection result;

[0041] The training of the end-to-end three-dimensional instance change detection model comprises:

[0042] ​The end-to-end three-dimensional instance change detection model based on deep learning is constructed, including: a multi-task shared feature encoder for extracting encoded basic features of input double-time point cloud data; a feature processing double-branch including a three-dimensional instance segmentation branch for optimizing change features, processing three-dimensional instance segmentation results obtained from the encoded basic features, and a three-dimensional change detection branch guided by instance features, processing three-dimensional change detection results output from the encoded basic features; and an information fusion module for fusing the three-dimensional change detection results and the three-dimensional instance segmentation results to obtain three-dimensional instance change detection results.

[0043] The training data set is constructed for model training, and the trained end-to-end three-dimensional instance change detection model is output.

[0044] Further, the multi-task shared feature encoder is a twin network structure, the multi-task shared feature encoder is a twin network structure, and the single network structure is a multi-level down-sampling structure including a plurality of feature encoding modules, the plurality of feature encoding modules are connected in sequence, input single-time point cloud data, and output encoded basic features of single-time point cloud data at different resolutions, the twin network structure processes single-time point cloud data at different times in double-time point cloud data respectively, and outputs encoded basic features of double-time point cloud data.

[0045] As a preferred embodiment, the twin network structure shares weights, which can reduce the memory occupancy rate of the model structure and improve the utilization rate of data.

[0046] Further, the feature encoding module includes a sampling layer, a feature extraction layer and a pooling layer; the sampling layer uses a grid down-sampling method to perform feature point sampling operation on the input to generate a plurality of feature kernel points; the feature extraction layer includes a plurality of feature extraction structures, a single layer structure includes a KPConv module, a batch normalization operation and a ReLU activation operation, for a plurality of feature kernel points of the sampling layer, a local neighborhood is constructed using a KNN algorithm, the KPConv module extracts features according to the spatial distribution of points in the local field, and the extracted features are connected with the input features in residual connection to retain the original geometric information, and finally the batch normalization and ReLU activation operation are performed to obtain the feature extraction result of the layer; the pooling layer processes the feature extraction result of the last feature extraction structure, aggregates high-level feature information using a cross-step KPConv down-sampling, and outputs the encoded basic features.

[0047] Further, the three-dimensional instance segmentation branch includes an instance refinement module and an instance segmentation decoder, the point cloud encoded basic features are input into the instance segmentation decoder for step-by-step processing to obtain instance-specific features, the instance refinement module obtains mask information and category information of the instance based on the instance-specific features, the initial point cloud encoded basic features and the query to obtain the three-dimensional instance segmentation results, and the three-dimensional instance segmentation results of double-time point cloud are obtained by processing the three-dimensional instance segmentation results in sequence.

[0048] As a preferred embodiment, multiple instance refinement modules can be included in an instance segmentation branch, which are stacked together to calculate the instance mask information.

[0049] Further, the instance segmentation decoder is composed of multiple layers of instance decoding modules, the number of layers being one less than the number of layers of the feature encoder network structure. The input point cloud encoding basic features are input to the first layer of instance decoding module, and the processing is performed layer by layer. The last layer of instance decoding module outputs instance-specific features. Each layer of instance decoding module is composed of a convolution layer and an up-sampling layer. The convolution layer uses CNN convolution operation to transform the feature space, and the up-sampling layer uses the nearest neighbor point up-sampling method to restore the point cloud features.

[0050] Further, the instance refinement module is composed of an instance mask calculation module and an instance query refiner module. The initialized query and instance-specific features are input to the instance mask calculation module. The similarity score between points is obtained by cross-multiplication as the attention mask, and the filtered result is used as the mask information of the instance according to the preset attention mask threshold. The attention mask, instance-specific features, and query are input into the instance query refiner module. The query refiner is optimized using the Transformer structure, and the class information of the instance is output.

[0051] Further, the three-dimensional change detection branch is composed of multiple layers of change information extraction modules and multiple layers of change feature decoders. The number of layers of the change information extraction module is consistent with the number of layers of the feature encoder network structure, and the number of layers of the change feature decoder is one more than the number of layers of the feature encoder network structure.

[0052] The change information extraction module is used to extract the change-specific features of the encoding basic features, and the change-specific features of the current layer are extracted and output layer by layer. In the change information extraction module, the KNN algorithm is used to match the neighborhood relationship of the double-time point cloud, and then the feature difference is extracted by calculating the feature difference to obtain the change-specific features. The change feature decoder is composed of a convolution layer and an up-sampling layer. The convolution layer uses CNN convolution operation to transform the feature space, and the up-sampling layer uses the nearest neighbor point up-sampling method to restore the point cloud features.

[0053] The first layer of change feature decoder takes the double-time point cloud encoding basic features and the change-specific features output by the first layer of change information extraction module as input, and outputs the decoding result. The input of the subsequent change feature decoder is the change-specific features output by the corresponding layer of change information extraction module and the decoding result of the previous layer of change feature decoder. The processing is performed level by level, and the output of the last layer of change feature decoder is fused with the output of the last layer of change feature decoder to obtain the final three-dimensional change detection result.

[0054] Further, the feature fusion module detects the changed target instance point cloud according to the three-dimensional change detection result and the three-dimensional instance segmentation result, and outputs a three-dimensional change detection result and a three-dimensional instance segmentation result fusion result of the target instance point cloud as the three-dimensional instance change detection result.

[0055] Further, the loss value in the training process of the end-to-end three-dimensional instance change detection model is a comprehensive loss value, wherein the comprehensive loss value is a weighted sum result of the three-dimensional instance segmentation result loss value and the three-dimensional change detection result loss value.

[0056] The present application is mainly based on a deep learning artificial neural network, fully considers the task characteristics of end-to-end three-dimensional instance change detection, and proposes a first end-to-end three-dimensional instance change detection method for point clouds based on deep learning. The method effectively couples three-dimensional change detection and three-dimensional instance segmentation in the model structure in the form of multi-task feature fusion, and strengthens the expression of instance change features. By using the characteristics of change heterogeneity and consistent invariance of change features in multiple time sequences, the instance segmentation performance of the model is strengthened, and by using the explicit mask characteristics of instance features, the model is guided to focus on the change of foreground target instances and suppress the influence of background non-target object changes on the model, so that the two tasks promote each other to improve, and the instance is segmented at the same time, and whether each instance changes is mined, so that efficient and accurate city three-dimensional instance change detection is realized.

[0057] The method provided by the present application can be realized by computer software technology, and the whole is referred to Figure 2 An end-to-end instance-level three-dimensional change detection method for point clouds, the instance is described with respect to the flow of two time sequence input point clouds, and the specific embodiment takes the Pytorch programming environment as an example, and the specific implementation is as follows:

[0058] Step 1, a multi-task shared feature encoder is constructed. The input data of the multi-task shared feature encoder is double time sequence point cloud, and a twin network structure is adopted to extract the encoding basic features of the double time sequence point cloud.

[0059] The three-dimensional instance change detection is composed of three-dimensional change detection task and three-dimensional instance segmentation task, and these tasks all depend on the data characteristics of the input point cloud. Therefore, a multi-task shared feature encoder is constructed to extract the encoding basic features containing the geometric morphological features and attribute features of the input data. The features describe the ubiquitous basic features of the double time sequence point cloud, mainly including geometric morphological features, texture material attribute features, etc., such as density, edge, corner, convex point, etc. The encoding basic features are shared among multiple tasks, realizing high adaptability of the model in multiple tasks.

[0060] Further, the multi-task shared feature encoder is a twin network structure, different network structures process different time single time point cloud data in the input double time point cloud data, and extract point cloud encoding basic features, wherein a single network structure is a multi-level down-sampling structure and is composed of multiple feature encoding modules. The multi-level down-sampling structure reduces the point cloud density layer by layer and expands the receptive field, thereby obtaining encoding basic features of different resolutions. The feature encoding module is the main functional module of feature extraction, which is composed of three functional modules, namely, a sampling layer, a feature extraction layer and a pooling layer. The sampling layer uses a grid down-sampling method to perform feature point sampling operation on the input point cloud to generate multiple feature kernel points as the reference position for subsequent feature convolution. The feature extraction layer is composed of multiple KPConv modules, batch normalization operation and ReLU activation operation. In the feature extraction layer, first, a local neighborhood is constructed for each feature kernel point from the sampling layer using the KNN algorithm; then based on the constructed local neighborhood, the KPConv module is used to extract features according to the spatial distribution of the input points; finally, the extracted features are connected with the input features in residual connection, thereby relieving the problem of gradient disappearance. The pooling layer uses a stride KPConv to realize down-sampling, gradually reduces the number of points and aggregates high-level feature information. After processing by multiple feature encoding modules, multiple encoding basic features of different resolutions are output, which will be input into the dedicated module of different tasks for mining of task-specific features, and through model back propagation, the feature encoder is guided to extract features beneficial to change detection and instance segmentation, so that the basic features contain rich implicit instance information and implicit change information, thereby improving the accuracy of subsequent tasks.

[0061] Specifically, given double time point cloud data of the same region as basic data. Before model processing, the input data is obtained by using cylinder sampling. The sampling radius of the cylinder sampling varies according to the density of different data sets. The sampling radius used is 30. The embodiment takes a five-level down-sampling structure as an example, and the single network structure of the multi-task shared feature encoder is shown in FIG. 1. Figure 3 As shown in FIG. 1, the feature dimension of the input data is 64, and the scale of the initial grid down-sampling is 1. After each feature encoding module is processed, the feature dimension of the data is expanded by 2 times, and the scale of the grid down-sampling is also expanded by 2 times. Therefore, the feature dimension of the entire encoder is 64, 128, 256, 512, 1024, and the down-sampling scale is 1, 2, 4, 8, 16. When the neighborhood is constructed, the maximum number of field points is set as an upper limit, and different threshold values are set for each layer, which are 25, 30, 38, 38 and 38 respectively.

[0062] Step 2, a three-dimensional instance segmentation branch optimized for change features is constructed.

[0063] The descriptive and specific task of the basic feature encoding obtained in step 1 is weak, and task-specific processing is still needed. Therefore, a three-dimensional instance segmentation branch with change feature cooperation is constructed to process the basic feature encoding to obtain instance-specific features containing change features, and based on the spatial heterogeneity characteristics of change features, the model is guided to achieve more accurate three-dimensional instance segmentation. The three-dimensional instance segmentation branch is the main functional module of the entire end-to-end three-dimensional instance change detection method for instance segmentation tasks, and its goal is to convert the basic feature encoding output by the multi-task shared feature encoder into instance-specific features for instance segmentation tasks, thereby obtaining three-dimensional instance information.

[0064] Further, the branch mainly consists of an instance segmentation decoder and an instance refinement module:

[0065] The instance segmentation decoder converts the basic feature encoding in the feature space to align with the instance information to obtain instance-specific features. The instance segmentation decoder consists of multiple instance decoding modules, which are processed layer by layer. Each layer of the instance decoding module is composed of a convolution layer and an upsampling layer, the convolution layer uses CNN convolution operation to transform the feature space, and the upsampling layer uses the nearest neighbor point upsampling method to restore the point cloud to a larger scale. The features processed by the instance decoder have been converted from point cloud basic features to instance-specific features, which can be used for the derivation of three-dimensional instance segmentation. However, this instance-specific feature is still relatively rough and needs to be input into the instance refinement module for further refinement, so as to achieve more accurate instance segmentation.

[0066] The instance refinement module consists of an instance query refiner module and an instance mask calculation module. The instance mask calculation module is responsible for calculating the mask of the instance contained in the input data, and is the main module for generating instance information. Its input data is the initialization query and the instance-specific feature, by cross-multiplying the instance-specific feature and the initialization query, the similarity score between points is obtained, and according to the score, the threshold judgment method is used to output the attention mask and the mask information of the instance. In the instance query refiner module, the attention mask output by the instance mask calculation module, the instance-specific feature, and the query are given as input data, and the Transformer structure is used to realize the optimization of the query by using multiple layers of attention operation, so that the query is aligned with the feature of the instance to obtain the class information of the instance. Finally, through the three-dimensional instance segmentation branch, the class information and the mask information of the instance are output. From the processing flow of this module, it can be seen that the input data of multiple modules uses the basic feature encoding, and these features contain change information, which can implicitly guide the model to pay attention to the heterogeneity before and after the change, so as to realize accurate instance segmentation with more rich feature information.

[0067] Specifically, the input data is multi-layer basic feature encoding and a defined query. As shown in FIG. 4, the input data is first input into the feature encoder to obtain the basic feature encoding, and then the basic feature encoding is input into the three-dimensional instance segmentation branch to obtain the class information and the mask information of the instance. Figure 4As shown, the instance decoder has one less layer than the feature encoder, which is 4 layers. The input feature dimension of each layer of the instance decoder is 3072, 1024, 512, 256, and the output dimension is 512, 256, 128, 64. An instance segmentation branch can include multiple instance refinement modules, which will be stacked together to calculate the instance mask information. In this embodiment, 5 layers of instance refinement modules are used. The number of instance queryers is defined as 160, and the initialization method uses the core point representation method, which uses the farthest point sampling method to collect multiple representative core points and uses the features of these points to initialize the instance queryer. In the instance queryer refinement module, split-head cross attention and self-attention are used, and the split-head cross attention uses 8 attention heads.

[0068] Step 3, build an instance feature guided three-dimensional change detection branch.

[0069] Consistent with the idea of step 2, the point cloud base feature of step 1 does not have the ability to describe the characteristics of the change detection task. Therefore, an instance feature collaborative three-dimensional change detection branch is constructed, which uses change-specific features containing instance features to guide the model to focus on the changes of the target instance object and suppress the influence of background noise data on the model, thereby achieving more accurate three-dimensional change detection.

[0070] Further, the three-dimensional change detection branch is the main functional module of the entire model for realizing change detection, and its main purpose is to convert the encoded base feature into change information in the multi-temporal point cloud, which is mainly composed of a change information extraction module and a change feature decoder:

[0071] The change information extraction module is the main functional module for converting the encoded base feature into change-specific features, and its input data is the encoded base feature. The change information extraction module first uses the KNN algorithm to obtain the neighborhood points of one time series point cloud in another time series point cloud in the encoded base feature, i.e., calculates the correspondence of the double time series point cloud. Then, based on these correspondences, the feature difference method is used to calculate the feature difference of the double time series point cloud, realizing the specific conversion of the encoded base feature of the point cloud to the change-specific feature.

[0072] The change feature decoder is a further refinement of the change-specific feature, which converts the change-specific feature into specific change information, and is composed of multiple change decoding modules. The composition of the change decoding module is consistent with that of the instance decoding module, which is composed of convolution operation and neighborhood upsampling. The convolution layer uses CNN convolution operation to transform the feature space, and the upsampling layer uses the nearest neighbor point upsampling method to restore the point cloud feature.

[0073] The three-dimensional change detection branch finally outputs the three-dimensional change detection result of the double time-series point cloud, and in the three-dimensional change detection branch, the input data is the basic point cloud feature containing instance information, which can implicitly guide the model to focus on the change of the target instance, thereby improving the change detection accuracy of the model.

[0074] Specifically, as shown in Figure 5 The number of change information extraction modules is consistent with the number of feature encoding modules, both of which are 5. In calculating the neighborhood relationship, the KNN algorithm is adopted, and the nearest point is directly used as the representative point. In calculating the feature difference, the algebraic difference method is adopted in this embodiment, that is, the features are directly subtracted to obtain the change feature. The change feature decoder is consistent with the instance feature decoder, which is also 4 layers. The input dimension is 3072, 1024, 512, 256, and the output dimension is 512, 256, 128, 64. It is worth noting that the input data is double time-series data, and each time-series data needs to be processed once through the three-dimensional change detection branch, and the entire three-dimensional change detection branch shares the weight parameters, that is, the same weight is still used for the double time-series, so that the utilization rate of data is improved while reducing the amount of model parameters.

[0075] Step 4, forming an end-to-end three-dimensional instance change detection model deeply coupled with instances and changes.

[0076] Steps 2 and 3 will respectively output the three-dimensional change detection result and the three-dimensional instance segmentation result. In order to evaluate the comprehensive performance of the model, the above structure and module output structure are fused, and a multi-task coupled end-to-end three-dimensional instance change detection model is proposed: by fusing the three-dimensional change detection result and the three-dimensional instance segmentation result, the three-dimensional instance change detection result is formed, and finally the end-to-end three-dimensional instance change detection is realized.

[0077] Further, accurate three-dimensional instance information and three-dimensional change detection results can be obtained through steps 2 and 3. In order to obtain comprehensive three-dimensional instance change information, it is also necessary to fuse these results. The strategy adopted is that the feature fusion module detects the target instance point cloud that has changed according to the three-dimensional change detection result and the three-dimensional instance segmentation result, performs information fusion of the three-dimensional change detection result and the three-dimensional instance segmentation result on the target instance point cloud, and outputs the fusion result to obtain the three-dimensional instance change information.

[0078] Further, in addition to fusing the output results, the loss value training method of the model also needs to be integrated. Three-dimensional instance segmentation and three-dimensional change detection are two different tasks, both of which contain independent loss values. If only one loss value is trained separately, the performance of the model will be tilted towards a single task, and the comprehensive three-dimensional instance change detection performance cannot be achieved. Therefore, a three-dimensional instance change detection comprehensive loss value calculation method is proposed, which obtains a comprehensive performance loss value by weighted summation of the three-dimensional instance segmentation and three-dimensional change detection loss values, so as to ensure that the model can simultaneously learn the features of multiple tasks.

[0079] The specific instance segmentation result is shown in Figure 6 From the figure, it can be seen that by fusing the three-dimensional instance segmentation result and the three-dimensional change detection result, the model can effectively output the three-dimensional instance change comprehensive information, and can achieve accurate three-dimensional instance change detection. When calculating the loss value, the loss value of the three-dimensional instance change detection is composed of the DICE loss of the instance mask information and the binary cross entropy, and the loss value of the three-dimensional change detection is composed of the binary cross entropy of the change information. According to the difference in loss value size and gradient of the two different tasks, the weight of the three-dimensional instance change detection loss value is set to 50, and the weight of the three-dimensional change detection loss value is set to 1, which can effectively realize the cooperative training of multi-task loss value.

[0080] The implementation basis of each embodiment of the present application is realized by the programmed processing of a system with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of the above embodiments, the embodiments of the present application provide a point cloud oriented end-to-end instance level three-dimensional change detection system, which is used to execute a point cloud oriented end-to-end instance level three-dimensional change detection method in the above method embodiments.

[0081] Referring to Figure 7The system comprises: a data acquisition and arrangement module, which acquires three-dimensional point cloud data of target objects in a target area and arranges the three-dimensional point cloud data into double-time-series point cloud data; a three-dimensional instance change detection module, which processes the double-time-series data and outputs a three-dimensional instance change detection result; and a training method of the end-to-end three-dimensional instance change detection model, which comprises the following steps: constructing an end-to-end three-dimensional instance change detection model based on deep learning, including a multi-task shared feature encoder for extracting encoded basic features of input double-time-series point cloud data, a feature processing double-branch including a three-dimensional instance segmentation branch optimized for change features for processing three-dimensional instance segmentation results obtained from the encoded basic features and a three-dimensional change detection branch guided by instance features for processing three-dimensional change detection results output from the encoded basic features; an information fusion module for fusing the three-dimensional change detection results and the three-dimensional instance segmentation results to obtain the three-dimensional instance change detection results; and constructing a training data set for model training to output the trained end-to-end three-dimensional instance change detection model.

[0082] It should be noted that the system embodiments provided by the present application are used to implement the methods in the above method embodiments, and are also used to implement the methods in other method embodiments provided by the present application, the difference is only that the corresponding function modules are set, and the principle is basically the same as that of the above system embodiments provided by the present application, as long as the person skilled in the art improves the modules in the above system embodiments on the basis of the above system embodiments, refers to the specific technical solutions in other method embodiments, obtains the corresponding technical means by combining technical features, and the technical solutions composed of these technical means, on the premise of ensuring the practicability of the technical solutions, the corresponding system class embodiments are obtained, which are used to implement the methods in other method class embodiments.

[0083] In summary of the above embodiments, the high-speed update of the city puts higher requirements on the accurate management of the city. The fine-grained three-dimensional instance change detection supervision of each ground object instance is the main method to achieve this goal. In order to solve the problem of large-scale city end-to-end three-dimensional instance-level change detection, the present application discloses a point cloud-oriented end-to-end instance-level three-dimensional change detection method. The method effectively couples the three-dimensional instance segmentation task and the three-dimensional change detection task in a unified model structure, uses instance features to guide change detection to focus on the change of target instance objects, and uses change features to enhance the instance segmentation performance of the model, so that the two tasks guide and promote each other in the model structure, and efficient end-to-end three-dimensional instance change detection is realized.

[0084] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A deep learning-based end-to-end 3D instance change detection method for point clouds, characterized by: include: Obtain three-dimensional point cloud data of the target object in the area to be measured and organize it into dual-time series point cloud data; Input the dual-time point cloud data into the trained end-to-end 3D instance change detection model to obtain the 3D instance change detection results; The training of the end-to-end 3D instance change detection model includes: An end-to-end 3D instance change detection model is constructed based on deep learning, including: a multi-task shared feature encoder for extracting the encoded basic features of the input dual-time series point cloud data; a dual feature processing branch, including a change feature-optimized 3D instance segmentation branch, which processes the encoded basic features to obtain 3D instance segmentation results, and an instance feature-guided 3D change detection branch, which processes the encoded basic features to output 3D change detection results; and an information fusion module, which fuses the 3D change detection results with the 3D instance segmentation results to obtain the 3D instance change detection results. Construct a training dataset for model training and output the trained end-to-end 3D instance change detection model.

2. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 1, characterized in that: The multi-task shared feature encoder is a twin network structure, and the single network structure is a multi-level downsampling structure, including multi-layer feature encoding modules. The multi-layer feature encoding modules are connected in sequence, input single-time sequence point cloud data, and output the encoding basic features of single-time sequence point cloud data of different resolutions. The twin network structure processes the single-time sequence point cloud data of different time sequences in the dual-time sequence point cloud data respectively, and outputs the encoding basic features of the dual-time sequence point cloud data.

3. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 2, characterized in that: The feature encoding module includes a sampling layer, a feature extraction layer, and a pooling layer; the sampling layer uses a grid downsampling method to perform feature point sampling operations on the input to generate multiple feature core points; The feature extraction layer contains several layers of feature extraction structures. The single-layer structure includes the KPConv module, batch normalization operation, and ReLU activation operation. For the feature kernel points of multiple sampling layers, the KNN algorithm is used to construct the local neighborhood. The KPConv module extracts weighted features based on the spatial distribution of points in the local area and performs a residual connection with the input features to retain the original geometric information. Finally, batch normalization and ReLU activation operations are performed to obtain the feature extraction results of this layer. The pooling layer processes the feature extraction results of the last feature extraction structure, uses strided KPConv downsampling to aggregate high-level feature information, and outputs the encoded basic features.

4. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 1, characterized in that: The three-dimensional instance segmentation branch includes an instance refinement module and an instance segmentation decoder. The basic features of the point cloud coding are input into the instance segmentation decoder for step-by-step processing to obtain instance-specific features. The instance refinement module obtains the mask information and category information of the instance as the three-dimensional instance segmentation result based on the instance-specific features, the initial point cloud coding basic features and the query.

5. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 4, characterized in that: The instance segmentation decoder is composed of multiple layers of instance decoding modules connected in sequence, where the number of layers is the number of layers of the feature encoder network structure minus one. The basic features of the point cloud encoding are input to the first-layer instance decoding module, and processed layer by layer. The last-layer instance decoding module outputs instance-specific features. Among them, each layer of the instance decoding module is composed of a convolutional layer and an upsampling layer. The convolutional layer uses CNN convolution operation to transform the feature space, and the upsampling layer uses the nearest neighbor upsampling method to restore the point cloud features.

6. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 4, characterized in that: The instance refinement module consists of an instance mask calculation module and an instance query refinement module. The initialized query and instance-specific features are input to the instance mask calculation module. The similarity score between points is obtained by cross product as the attention mask, and the result is filtered according to the preset attention mask threshold. The filtered result is used as the mask information of the instance. The attention mask, instance-specific features, and query are input into the instance query refinement module, and the query is optimized using the Transformer structure to output the category information of the instance.

7. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 1, characterized in that: The three-dimensional change detection branch is composed of a multi-layer change information extraction module and a multi-layer change feature decoder. The number of layers of the change information extraction module is consistent with the number of layers of the feature encoder network structure, and one more than the number of layers of the change feature decoder. The change information extraction module is used to extract the change-specific features of the encoding basic features, extracting and outputting the change-specific features of this layer layer by layer. In the change information extraction module, the KNN algorithm is used to match the point clouds of the dual time series to extract the neighborhood relationship, and then the feature difference is calculated to extract the feature difference to obtain the change-specific features. The change feature decoder consists of a convolutional layer and an upsampling layer. The convolutional layer uses the CNN convolution operation to transform the feature space, and the upsampling layer uses the nearest neighbor upsampling method to restore the point cloud features. The first-layer change feature decoder takes the dual-time point cloud coding basic features and the change-specific features output by the first-layer change information extraction module as input, and outputs the decoding result. The input of subsequent change feature decoders is the change-specific features output by the change information extraction module of the corresponding layer and the decoding result of the previous layer change feature decoder. The processing is carried out step by step, and the output of the last layer change feature decoder is spliced ​​and fused with the output of the last layer change feature decoder to obtain the final three-dimensional change detection result.

8. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 1, characterized in that: The feature fusion module detects the changed target instance point cloud according to the 3D change detection result and the 3D instance segmentation result, and outputs the fusion result of the 3D change detection result and the 3D instance segmentation result of the target instance point cloud as the 3D instance change detection result.

9. The end-to-end 3D instance change detection method for point cloud based on deep learning according to claim 1, characterized in that: The loss value during the training process of the end-to-end 3D instance change detection model is a comprehensive loss value, wherein the comprehensive loss value is a weighted sum of the 3D instance segmentation result loss value and the 3D change detection result loss value.

10. A deep learning-based end-to-end 3D instance change detection system for point clouds, characterized by: include: The data acquisition and organization module collects the three-dimensional point cloud data of the target object in the target area and organizes it into dual-time series point cloud data; 3D instance change detection module, which processes dual time series data and outputs 3D instance change detection results; The training of the end-to-end 3D instance change detection model includes: An end-to-end 3D instance change detection model is constructed based on deep learning, including: a multi-task shared feature encoder for extracting the encoded basic features of the input dual-time series point cloud data; a dual feature processing branch, including a change feature-optimized 3D instance segmentation branch, which processes the encoded basic features to obtain 3D instance segmentation results, and an instance feature-guided 3D change detection branch, which processes the encoded basic features to output 3D change detection results; and an information fusion module, which fuses the 3D change detection results with the 3D instance segmentation results to obtain the 3D instance change detection results. Construct a training dataset for model training and output the trained end-to-end 3D instance change detection model.

Images