Multi-mode building height change detection method for process decoupling

Abstract

The invention discloses a multi-modal building height change detection method based on process decoupling, and the method comprises the steps: constructing a building height change detection training data set through constructing a multi-modal cross-dimensional building height change detection model, and employing a digital ground model and image data; and a multi-modal cross-modal feature graph is obtained by using the multistage twin convolutional network and is input into a multi-modal Gaussian guided aggregation module to obtain final multi-scale cross-modal fusion features, the final multi-scale cross-modal fusion features are input into a decoding module for decoding output, and finally model network parameters are corrected and a model is trained. And the trained model is obtained to realize cross-modal quantitative detection of height changes of various types of buildings. According to the method, multi-modal image fusion is carried out by using variational gauss of multi-modal features and the change diagram, the detection problem of complex changes such as dismantling and then building in urban building changes is solved by a prediction mode of process decoupling, and compared with double-time-phase three-dimensional detection, the method has the advantages of simple preprocessing, low detection cost, flexible detection frequency and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a process-decoupled multi-modal building height change detection method. Background Art

[0002] Change detection refers to the use of multi-temporal data collected at different times, defining change categories and attributes according to certain research objectives, and distinguishing each sub-data pair composed of multi-temporal data into two sets of changes and invariants based on pattern recognition or machine vision methods. Change detection technology is widely used in research fields such as remote sensing images and natural image scene understanding, and plays a huge role in real-world applications closely related to humans, such as urban evolution monitoring, land use analysis, deforestation estimation, and glacier evolution.

[0003] At present, the mainstream remote sensing image change detection algorithms are divided into two categories according to the data used. One category belongs to single-modal change detection, that is, using dual-phase two-dimensional images (such as satellite images, aerial images) or three-dimensional data (such as digital ground models, point cloud data) to achieve. This type of method is widely studied and relatively mature. However, the use of dual-phase three-dimensional data is limited by slow data update speed and long preprocessing time, resulting in low detection frequency and high cost. There may also be data missing, and the practicality is poor. The second category is multimodal change detection algorithms. Existing algorithms often introduce complementary modal features such as infrared and SAR (Synthetic Aperture Radar) images on the basis of optical images, so that change detection can be carried out under conditions of low light intensity and cloud interference, which greatly improves the environmental adaptability of change detection algorithms. However, this type of algorithm is limited to two-dimensional change detection, that is, the change is limited to the qualitative detection level of a single or multiple categories; in terms of introducing three-dimensional data to quantitatively detect high-dimensional changes, such as building height changes, there is still a lack of relevant research. Due to the huge differences in modal features, the fusion and joint change detection of multimodal change detection is still a difficult problem in the field. At the same time, the current annotation method for urban building change detection uses a single category code bit, that is, it only describes the category of the change, such as no change, new building, and demolition. This will cause category code overlap in complex urban changes, resulting in unclear descriptions.

[0004] In summary, the current mainstream remote sensing image change detection methods have two problems in the application scenario of quantitative height change estimation, including: 1) Single-modal three-dimensional change detection has problems such as high data acquisition and processing costs, low detection frequency, and data missing; 2) Multimodal change detection is limited to qualitative two-dimensional change detection, and there is currently a lack of research on the introduction of three-dimensional data to achieve quantitative change detection; 3) The fusion of multimodal features and joint change detection are difficult problems in the field; 4) The current urban building change detection annotation has process coupling, which makes the description of the change process unclear. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a process-decoupled multimodal building height change detection method, which uses digital terrain model elevation data and two-dimensional image data to realize quantitative height change detection of new construction and demolition of urban buildings.

[0006] The technical solution adopted by the present invention is: a process-decoupled multi-modal building height change detection method, the specific steps are as follows:

[0007] S1. Construct a multi-modal cross-dimensional building height change detection model, including: input module, multi-level twin convolutional network, multi-modal Gaussian guided aggregation module, and decoding module;

[0008] S2, input digital terrain models and image data of different periods, regardless of the previous and the next, into the input module, perform semi-automatic annotation based on the dual-phase digital terrain model DSM difference map and the building segmentation map, realize process-decoupled building category and height change annotation, and obtain a building height change detection training data set;

[0009] The building height change detection training data set includes: DSM data of the previous phase, image data of the next phase, and the category change annotation map GT of the two phases. class-pre With GT class-post , and the building height change annotation map GT height .

[0010] S3, inputting the front-phase digital terrain model and the back-phase image data into a multi-level twin convolutional network with shared parameters in parallel to obtain a multi-modal cross-modal feature map, namely, DSM height features and image features;

[0011] S4, based on the multimodal cross-modal feature map obtained in step S3, inputting the multimodal Gaussian guided aggregation module to perform height map guided multimodal feature fusion to obtain the final multi-scale cross-modal fusion feature;

[0012] Based on the DSM height features and image features of the corresponding scales obtained in step S3, they are respectively subjected to variational Gaussianization, and the KL divergence loss of the feature layer distribution parameters and the standard normal distribution is calculated. Then, random resampling is performed using the standard normal distribution, and then multimodal feature fusion guided by the height map is performed, that is, the multimodal fusion features are used to predict the height change, and the average bi-norm loss is calculated with the height change annotation map; for the multi-scale multimodal fusion features, the adjacent scales are first upsampled and fused, and then upsampled step by step to the maximum scale to obtain the final multi-scale cross-modal fusion features.

[0013] S5, based on the final multi-scale cross-modal fusion features obtained in step S4, input them into the decoding module, use two groups of three-channel 1×1 convolutions and one group of single-channel 1×1 convolutions for decoding and output, and obtain a three-channel category change prediction map representing the categories of the before and after phase changes, and a single-channel height change prediction map representing the height change;

[0014] S6. Calculate the cross entropy loss of the output three-channel category change prediction map and the category change annotation map, calculate the average two-norm loss of the output single-channel height change prediction map and the height change annotation map, calculate the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module, and use gradient back propagation to minimize the loss, and correct the network parameters of the multimodal cross-dimensional building height change detection model;

[0015] S7, training a multi-modal cross-dimensional building height change detection model, i.e., looping steps S3-S6, traversing all samples in the building height change detection training data set until the loss curve is smooth, and finally obtaining a trained building height change detection model that can be used for multi-modal cross-dimensional data;

[0016] S8. Based on the trained building height change detection model obtained in step S7, the digital ground model to be inspected and the corresponding image data are input, and a category change prediction map representing the construction and demolition of buildings and a corresponding quantitative height change prediction map are output to achieve cross-modal quantitative detection of height changes of multiple categories of buildings.

[0017] Furthermore, the step S2 is specifically as follows:

[0018] Changes are annotated for multi-modal cross-dimensional data pairs consisting of digital terrain models (DSM) and image data at different times. That is, the annotation process uses the dual-phase DSM difference map and the building segmentation map to pre-annotate the changed area, and combines the definition of building height changes to achieve semi-automatic annotation.

[0019] First, the digital terrain model H post With H pre Take the difference, that is, H post -H pre Get the height difference map H diffMerge the building areas of the previous and next phases respectively to obtain the merged image S of the building areas of the two phases. building-pre With S building-post .

[0020] Then the height difference map H diff Merge map with building area S building-pre With S building-post The two phases are intersected to obtain a preliminary building area height change map, and the threshold method is used to mark the building area height change map to obtain a preliminary building category change marking map.

[0021] Among them, the threshold method classifies building changes with a height change ΔH greater than a set threshold h as newly built, classifies building changes with a height change ΔH less than 0 as demolished, and classifies building changes with a height change ΔH greater than or equal to 0 but less than or equal to the set threshold h as no change.

[0022] The building category change annotation map is set with 0, 1, and 2 to represent no building, building, and new building, respectively. Then, the two phases are coded according to the building area height change map of the two phases, that is, two coding positions are used to describe a total of 6 types of ground feature coverage changes in the two phases, including: in areas with demolition changes, 1-0 is used to indicate that there was a building in the previous phase, and 1-2 is used to indicate that there was a building in both phases; in areas with new construction changes, 0-1 is used to indicate that there is a building in the next phase, and 1-2 is used to indicate that there is a building in both phases; 1-1 is used to represent building areas with no building changes; and 0-0 is used to represent ground areas with no building changes.

[0023] Finally, the preliminary building category change annotation map is manually adjusted, including: adjusting the discontinuous building outline edges and abnormal height changes caused by tree occlusion, to obtain the final building category change annotation map GT class-pre , perform XOR operation on the category change annotation graphs of the two phases, and then compare them with the building height change graph H diff Intersect and get the building height change annotation map GT height .

[0024] DSM data of the previous phase, image data of the next phase, and the category change annotation map of the two phases GT class-pre With GT class-post , and the building height change annotation map GT height , forming a building height change detection dataset.

[0025] Furthermore, the step S3 is specifically as follows:

[0026] In each training, the pre-phase DSM and post-phase image data are input into the multi-level twin convolutional network in parallel to obtain the multi-scale feature maps of the two modalities. The multi-modal features of any scale are derived from the features of the digital terrain model. Image characteristics constitute.

[0027] The multi-stage twin convolutional network includes Z-stage serially connected convolutional blocks, and convolutional blocks are used stage by stage to process two different modal inputs.

[0028] Among them, the number of convolution blocks connected in series in the multi-stage twin convolutional network is set according to actual conditions.

[0029] For image modality input Directly use Z-level concatenated convolution blocks to process and obtain multi-scale feature output Each convolution block consists of 3 to 5 serially connected 3×3 convolutional layers, batch normalization layers, and activation function layers. The output of the activation function layer is added to the input of the convolutional layer as a residual connection. A convolutional layer with a stride of 2 is used for downsampling before each level of input, and the scale of the next level in the Z-level output is 0.5 times that of the previous level.

[0030] The DSM modality input passes through a Z-level series of convolution blocks that share the input parameters of the imaging modality to obtain a multi-scale feature output.

[0031] Furthermore, the step S4 is specifically as follows:

[0032] Based on step S3, the previous phase DSM and the next phase image data are input into the multi-level twin convolutional network in parallel to obtain a multimodal cross-modal feature map. as well as After that (i=1,2,3,…Z), for two different modal features of any scale, variational Gaussianization is performed, that is, the mean and variance of the features are obtained by using 1×1 convolution mapping, and the expressions are as follows:

[0033]

[0034] in, represents the mean and variance of the DSM height feature, Represents the mean and variance of the image features. Then the Kullback-Leibler divergence of the distribution corresponding to these two sets of means and variances and the standard normal distribution is calculated, that is, the Kullback-Leibler divergence loss is used to constrain and The corresponding mean and variance satisfy the normal distribution, and the expressions are as follows:

[0035]

[0036] Among them, p1(x) represents or The distribution obeyed, p2(x) represents the expected approximation distribution. When p2(x) ~ N(0,1), the KL loss expression is as follows:

[0037]

[0038] Here, KL(p1||p2) represents the KL divergence loss of p1 and p2, which is a measure of the difference between one probability distribution and another. And σ and μ are replaced by or By minimizing the above KL loss, or The mean and variance satisfy the standard normal distribution. The distribution constraint loss expression of the i-th layer of multimodal features to be fused is defined as follows:

[0039]

[0040] Then use a standard normal distribution to resample the mean and variance after variational Gaussianization, and splice the resampled features according to the channel, that is, the two sets of means and variances obtained by the constraints Use another standard normal distribution ε to sample and add it to the original features before Gaussianization. The expression is as follows:

[0041]

[0042] Among them, ε~N(0,1), and Represents the final variational Gaussian modal enhancement feature. The enhanced features of the two modalities are concatenated by channel to obtain the modal fusion feature at this scale.

[0043] Fusion feature F i The building height change annotation map is used to guide aggregation as the cross-modal feature of the current scale level. The expression is as follows:

[0044]

[0045] in, Indicates channel-by-channel concatenation. Tanh maps the input to the range of [-1,1]. F i (i,j) represents the i-th row and j-th column of the joint feature, GT height (i,j) represents the i-th row and j-th column on the real height change graph. It represents the bi-norm loss between the height change prediction map and the height change annotation map output by the i-th level multimodal fusion feature; M represents the total number of rows of the annotation map, and N represents the total number of columns of the annotation map.

[0046] The length and width of the next-level feature map are 0.5 times that of the previous-level feature map. For cross-modal features of multiple scales, adjacent scales are first upsampled and fused, and then upsampled to the maximum scale step by step. That is, 2 times upsampling is used to add and fuse adjacent multi-scale feature maps to obtain a multi-scale feature map group. The expression is as follows:

[0047]

[0048] Among them, UP 2× Indicates 2x upsampling.

[0049] Then all scale features are upsampled to the first-level feature scale, and multi-scale feature maps are concatenated to obtain the final multi-scale cross-modal fusion feature F out , the expression is as follows:

[0050]

[0051] Furthermore, the step S5 is specifically as follows:

[0052] The final multi-scale cross-modal fusion feature F obtained in step S4 out , using two groups of three-channel 1×1 convolutions and one group of single-channel 1×1 convolutions, activated by softmax and tanh functions respectively, we get the category change prediction map and the height change prediction map, the expressions are as follows:

[0053] Pred class-pre =softmax(con1×1(F out ))

[0054] Pred class-post =softmax(con1×1(F out ))

[0055] Pred height =Tanh(con1×1(F out ))

[0056] Among them, the softmax function normalizes each point of the three-channel category change prediction map to the range of 0 to 1 according to the channel dimension. The first channel represents the probability of no building, the second channel represents the probability of having a building, and the third channel represents the probability of building a new building; the Tanh function normalizes the single channel output to the range of -1 to 1; Pred class-pre Indicates the prediction result of the previous phase, Pred class-postIndicates the prediction result of the next phase, Pred height Represents the height change prediction map.

[0057] Furthermore, the step S6 is specifically as follows:

[0058] Firstly, the cross entropy loss of the output three-channel category prediction map and the category change annotation map is calculated, the average two-norm loss of the single-channel height change prediction map and the height change annotation is calculated, and the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module are calculated. The weighted sum of the above losses is the final training loss. The training loss is minimized by gradient back propagation, and the network parameters of the multimodal cross-dimensional building height change detection model are corrected.

[0059] The cross entropy loss function is used as the cost function, and the calculation expression is as follows:

[0060]

[0061] Among them, GT class-pre (i,j) and GT class-post (i,j) represents the i-th row and j-th column on the two time phase category annotation maps, and its values ​​are 0, 1, and 2, representing no buildings, buildings, and new buildings, respectively; M represents the total number of rows in the annotation map, and N represents the total number of columns in the annotation map.

[0062] The two-norm loss is used as the height change cost function, and the calculation expression is as follows:

[0063]

[0064] The final training cost function is loss class-pre 、loss class-post 、loss height The weighted sum of is expressed as follows:

[0065]

[0066] Among them, λ1, λ2, λ3, and λ4 are constants greater than 0, representing the weight of each loss, which can be fine-tuned according to the actual application data set. By minimizing the loss, the final change detection model is obtained.

[0067] Beneficial effects of the present invention: The present invention discloses a process-decoupled multimodal building height change detection method, which constructs a multimodal cross-dimensional building height change detection model, uses digital ground models and image data to construct a building height change detection training data set, and then uses a multi-level twin convolutional network to obtain a multimodal cross-modal feature map to input the multimodal Gaussian guided aggregation module to obtain the final multi-scale cross-modal fusion feature, which is input into the decoding module for decoding output, and finally the model network parameters are corrected and the model is trained to obtain a trained model to achieve cross-modal quantitative detection of multi-class building height changes. The method of the present invention solves the problem of quantitative change detection of building height in the absence of multi-phase three-dimensional data, and uses the variational Gaussian and change map of multimodal features to fuse multimodal images. At the same time, its process-decoupled prediction method solves the detection problem of complex changes such as demolition first and then new construction in urban building changes. Compared with dual-phase three-dimensional detection, it has the advantages of simple preprocessing, low detection cost, and flexible detection frequency.

[0068] The method of the present invention can overcome the problem of lack of quantitative evaluation of height changes in existing two-dimensional category change detection methods, as well as the difficulties of data acquisition, slow speed, and high cost in dual-phase three-dimensional change detection methods; only the height data of a single phase and the image data of the corresponding area at another time are required to achieve simultaneous detection of multiple building categories and height changes. At the same time, the constructed detection method model is based on the convolution backbone to learn the feature context within each modality, and uses variational Gaussianization and height map guidance mechanisms to learn cross-modal feature enhancement representations, and finally uses a multi-level feature fusion method to decode and output the category and height changes separately. The method of the present invention solves the problem of quantitative height change detection of buildings, and does not require the use of height data of two phases. The introduced image modality data can effectively improve the detection frequency and flexibility, reduce the cost of quantitative height change detection, and its process decoupled annotation method is conducive to the description of complex urban building changes, and has strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 The present invention is a flowchart of a process-decoupled multi-modal building height change detection method.

[0070] Figure 2 This is a structural diagram of a multi-mode cross-dimensional building height change detection model in an embodiment of the present invention.

[0071] Figure 3 It is a schematic diagram comparing conventional change annotation and process-decoupled change annotation in an embodiment of the present invention.

[0072] Figure 4 This is a structural diagram of a multimodal Gaussian guided aggregation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0074] like Figure 1 As shown, a process decoupled multi-modal building height change detection method flow chart of the present invention, the specific steps are as follows:

[0075] S1, such as Figure 2 As shown, a multi-modal cross-dimensional building height change detection model is constructed, including: an input module, a multi-level twin convolutional network, a multi-modal Gaussian guided aggregation module MMGA, and a decoding module;

[0076] The input module is used to construct a training dataset for detecting building height changes based on digital terrain models and image data from different periods of time.

[0077] S2, input digital surface models and image data of different periods, regardless of the previous and the next, into the input module, perform semi-automatic annotation based on the dual-phase digital surface model (DSM) difference map and the building segmentation map, realize process-decoupled building category and height change annotation, and obtain a training data set for building height change detection;

[0078] The building height change detection training data set includes: DSM data of the previous phase, image data of the next phase, and the category change annotation map GT of the two phases. class-pre With GT class-post , and the building height change annotation map GT height .

[0079] S3, inputting the front-phase digital terrain model and the back-phase image data into a multi-level twin convolutional network with shared parameters in parallel to obtain a multi-modal cross-modal feature map, namely, DSM height features and image features;

[0080] S4, based on the multimodal cross-modal feature map obtained in step S3, inputting the multimodal Gaussian guided aggregation module to perform height map guided multimodal feature fusion to obtain the final multi-scale cross-modal fusion feature;

[0081] Based on the DSM height features and image features of the corresponding scales obtained in step S3, they are respectively subjected to variational Gaussianization, and the KL divergence loss of the feature layer distribution parameters and the standard normal distribution is calculated. Then, random resampling is performed using the standard normal distribution, and then multimodal feature fusion guided by the height map is performed, that is, the multimodal fusion features are used to predict the height change, and the average bi-norm loss is calculated with the height change annotation map; for the multi-scale multimodal fusion features, the adjacent scales are first upsampled and fused, and then upsampled step by step to the maximum scale to obtain the final multi-scale cross-modal fusion features.

[0082] S5, based on the final multi-scale cross-modal fusion features obtained in step S4, input them into the decoding module, use two groups of three-channel 1×1 convolutions and one group of single-channel 1×1 convolutions for decoding and output, and obtain a three-channel category change prediction map representing the categories of the before and after phase changes, and a single-channel height change prediction map representing the height change;

[0083] S6. Calculate the cross entropy loss of the output three-channel category change prediction map and the category change annotation map, calculate the average two-norm loss of the output single-channel height change prediction map and the height change annotation map, calculate the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module, and use gradient back propagation to minimize the loss, and correct the network parameters of the multimodal cross-dimensional building height change detection model;

[0084] S7, training a multi-modal cross-dimensional building height change detection model, i.e., looping steps S3-S6, traversing all samples in the building height change detection training data set until the loss curve is smooth, and finally obtaining a trained building height change detection model that can be used for multi-modal cross-dimensional data;

[0085] S8. Based on the trained building height change detection model obtained in step S7, the digital ground model to be inspected and the corresponding image data are input, and a category change prediction map representing the construction and demolition of buildings and a corresponding quantitative height change prediction map are output to achieve cross-modal quantitative detection of height changes of multiple categories of buildings.

[0086] In this embodiment, step S2 is specifically as follows:

[0087] Change annotation is performed on multi-modal cross-dimensional data pairs composed of digital surface models (DSM) and image data at different times. That is, the annotation process uses the dual-phase DSM difference map and the building segmentation map to pre-annotate the change area, and combines the definition of building height changes to achieve semi-automatic annotation. The schematic diagram of the comparison between conventional change annotation and the decoupled change annotation process of this embodiment is shown in Figure 2. Figure 3 As shown, Figure 3 (a) is a regular change mark, Figure 3 (b) is a change annotation for process decoupling. Figure 3 It can be seen that in conventional annotation, the process of demolishing buildings first and then rebuilding them is coupled with the pure process of demolishing and rebuilding buildings, and the description of the change process of land cover is not clear. In the change annotation of process decoupling of the method of the present invention, two coding positions are used to describe a total of 6 types of land cover changes in two phases, which clearly depicts the complex changes of demolishing first and then building in urban construction.

[0088] First, the digital terrain model H post With H pre Take the difference, that is, H post-H pre Get the height difference map H diff The building areas in the previous and next phases (the rough building area map obtained by elevation clustering in the digital terrain model) are merged to obtain the merged building area map S of the two phases. building-pre With S building-post .

[0089] Then the height difference map H diff Merge map with building area S building-pre With S building-post The two phases are intersected to obtain a preliminary building area height change map, and the threshold method is used to mark the building area height change map to obtain a preliminary building category change marking map.

[0090] Among them, the threshold method classifies building changes with a height change ΔH greater than a set threshold h as newly built, classifies building changes with a height change ΔH less than 0 as demolished, and classifies building changes with a height change ΔH greater than or equal to 0 but less than or equal to the set threshold h as no change.

[0091] The threshold value for height increase or decrease is set to h in order to filter out most noise height changes, and this value can be adjusted according to the specific application area and data scenario. In this embodiment, h=3m.

[0092] like Figure 3 As shown in (b), 0, 1, and 2 are set on the building category change annotation map to represent no building, building, and new building, respectively. Then, the two phases are coded according to the building area height change map of the two phases, that is, two coding positions are used to describe a total of 6 types of ground feature coverage changes in the two phases, including: in areas with demolition changes, 1-0 is used to indicate that there was a building in the previous phase, and 1-2 is used to indicate that there was a building in both phases; in areas with new construction changes, 0-1 is used to indicate that there was a building in the next phase, and 1-2 is used to indicate that there was a building in both phases; 1-1 is used to indicate a building area with no building changes; and 0-0 is used to indicate a ground area with no building changes.

[0093] Finally, the preliminary building category change annotation map is manually adjusted, including: adjusting the discontinuous building outline edges and abnormal height changes caused by tree occlusion, to obtain the final building category change annotation map GT class-pre , perform XOR operation on the category change annotation graphs of the two phases, and then compare them with the building height change graph H diff Intersect and get the building height change annotation map GT height .

[0094] DSM data of the previous phase, image data of the next phase, and the category change annotation map of the two phases GT class-pre With GT class-post, and the building height change annotation map GT height , forming a building height change detection dataset.

[0095] In this embodiment, step S3 is specifically as follows:

[0096] In each training, the front-phase DSM and the back-phase image data are input into the multi-level twin convolutional network in parallel to obtain the multi-scale feature maps of the two modalities (multi-modal cross-modal feature maps). The multi-modal features of any scale are derived from the features of the digital terrain model. Image characteristics constitute.

[0097] The multi-stage twin convolutional network includes Z-stage serially connected convolutional blocks, and convolutional blocks are used stage by stage to process two different modal inputs.

[0098] The number of convolution blocks connected in series in the multi-stage twin convolution network is set according to actual conditions. In this embodiment, Z=4.

[0099] like Figure 2 As shown, this embodiment inputs the image modality Directly use Z = 4 levels of convolution blocks in series to obtain multi-scale feature output Each convolution block consists of 3 to 5 serially connected 3×3 convolutional layers, batch normalization layers, and activation function layers. The output of the activation function layer is added to the input of the convolutional layer as a residual connection. A convolutional layer with a stride of 2 is used for downsampling before each level of input, and the scale of the next level in the Z=4 level output is 0.5 times that of the previous level.

[0100] The DSM modality input passes through a Z=4-level series convolution block that shares the input parameters with the image modality input to obtain a multi-scale feature output.

[0101] The structure of the multimodal Gaussian guided aggregation module is as follows: Figure 4 As shown, in this embodiment, the step S4 is specifically as follows:

[0102] Based on step S3, the previous phase DSM and the next phase image data are input into the multi-level twin convolutional network in parallel to obtain a multimodal cross-modal feature map. as well as After that (i=1, 2, 3, 4), for two different modal features of any scale, variational Gaussianization is performed, that is, the mean and variance of the features are obtained by using 1×1 convolution mapping, and the expressions are as follows:

[0103]

[0104] in, represents the mean and variance of the DSM height feature, Represents the mean and variance of the image features. Then the Kullback-Leibler divergence of the distribution corresponding to the two sets of means and variances and the standard normal distribution is calculated, that is, the Kullback-Leibler (KL) divergence loss is used to constrain and The corresponding mean and variance satisfy the normal distribution, and the expressions are as follows:

[0105]

[0106] Among them, p1(x) represents or The distribution obeyed, p2(x) represents the expected approximation distribution. When p2(x) ~ N(0,1), the KL loss expression is as follows:

[0107]

[0108] Here, KL(p1||p2) represents the KL divergence loss of p1 and p2, which is a measure of the difference between one probability distribution and another. And σ and μ are replaced by or By minimizing the above KL loss, or The mean and variance satisfy the standard normal distribution. The distribution constraint loss expression of the i-th layer of multimodal features to be fused is defined as follows:

[0109]

[0110] Then use a standard normal distribution to resample the mean and variance after variational Gaussianization to make the process differentiable, and splice the resampled features according to the channel, that is, the two sets of means and variances obtained by the constraints Use another standard normal distribution ε to sample and add it to the original features before Gaussianization. The expression is as follows:

[0111]

[0112] Among them, ε~N(0,1), and Represents the final variational Gaussian modal enhancement feature. The enhanced features of the two modalities are concatenated by channel to obtain the modal fusion feature at this scale.

[0113] Fusion feature F i The building height change annotation map is used to guide aggregation as the cross-modal feature of the current scale level. The expression is as follows:

[0114]

[0115] in, Indicates channel-by-channel concatenation. Tanh maps the input to the range of [-1,1]. F i (i,j) represents the i-th row and j-th column of the joint feature, GT height (i,j) represents the i-th row and j-th column on the real height change graph. It represents the bi-norm loss between the height change prediction map and the height change annotation map output by the i-th level multimodal fusion feature; M represents the total number of rows of the annotation map, and N represents the total number of columns of the annotation map.

[0116] The length and width of the next-level feature map are 0.5 times that of the previous-level feature map. For cross-modal features of multiple scales, adjacent scales are first upsampled and fused, and then upsampled to the maximum scale step by step. That is, 2x upsampling is used to add and fuse adjacent multi-scale feature maps, so that the shallow feature map contains more deep semantic information, and a multi-scale feature map group is obtained. The expression is as follows:

[0117]

[0118] Among them, UP 2× Indicates 2x upsampling.

[0119] Then all scale features are upsampled to the first-level feature scale, and multi-scale feature maps are concatenated to obtain the final multi-scale cross-modal fusion feature F out , the expression is as follows:

[0120]

[0121] In this embodiment, step S5 is specifically as follows:

[0122] The final multi-scale cross-modal fusion feature F obtained in step S4 out , using two sets of 1×1 convolutions with an output dimension of 3 dimensions and a set of 1-dimensional input dimension (2 sets of three-channel 1×1 convolutions and 1 set of single-channel 1×1 convolutions), activated by softmax and tanh functions respectively, we get the category change prediction map and the height change prediction map, the expressions are as follows:

[0123] Pred class-pre =softmax(con1×1(F out ))

[0124] Pred class-post =softmax(con1×1(F out ))

[0125] Pred height =Tanh(con1×1(F out))

[0126] Among them, the softmax function normalizes each point of the three-channel category change prediction map to the range of 0 to 1 according to the channel dimension. The first channel represents the probability of no building, the second channel represents the probability of having a building, and the third channel represents the probability of building a new building; the Tanh function normalizes the single channel output to the range of -1 to 1; Pred class-pre Indicates the prediction result of the previous phase (the coverage of the objects in the previous phase), Pred class-post Indicates the prediction result of the next phase (the coverage of the ground objects in the next phase), Pred height Represents the height change prediction map.

[0127] In this embodiment, step S6 is specifically as follows:

[0128] Cross entropy loss and average bi-norm loss are used to measure the difference between the predicted image and the labeled image, and the network parameters are updated to minimize the difference.

[0129] Firstly, the cross entropy loss of the output three-channel category prediction map and the category change annotation map is calculated, the average two-norm loss of the single-channel height change prediction map and the height change annotation is calculated, and the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module are calculated. The weighted sum of the above losses is the final training loss. The training loss is minimized by gradient back propagation, and the network parameters of the multimodal cross-dimensional building height change detection model are corrected.

[0130] The cross entropy loss function is used as the cost function, and the calculation expression is as follows:

[0131]

[0132] Among them, GT class-pre (i,j) and GT class-post (i,j) represents the i-th row and j-th column on the two time phase category annotation maps, and its values ​​are 0, 1, and 2, representing no buildings, buildings, and new buildings, respectively; M represents the total number of rows in the annotation map, and N represents the total number of columns in the annotation map.

[0133] The two-norm loss is used as the height change cost function, and the calculation expression is as follows:

[0134]

[0135] The final training cost function is loss class-pre 、loss class-post 、loss height The weighted sum of is expressed as follows:

[0136]

[0137] Among them, λ1, λ2, λ3, and λ4 are constants greater than 0, representing the weight of each loss, which can be fine-tuned according to the actual application data set. In this embodiment, the values ​​are 1, 1, 0.1, and 0.1 respectively. By minimizing the loss, the final change detection model is obtained.

[0138] In summary, the method of the present invention, at the data level, uses the multi-dimensional cross-modal data of the single-phase digital ground model and the single-phase image data, which greatly alleviates the problem of the difficulty in obtaining height data in the process of quantitative height change detection, and the introduction of image data effectively improves the flexibility of detection frequency and reduces the cost of quantitative height change detection; in the annotation stage, the building area classification map is introduced to realize semi-automatic auxiliary annotation, and the process decoupled change annotation method is used to solve the description of the change process of complex urban buildings; in the training stage, a parameter sharing backbone based on the convolution module is constructed to efficiently learn the feature relationship within the modality, and the variational Gaussianization and height map guidance mechanism are introduced to learn the cross-modal association of two different modal features, through the semantic and height change decoding of feature multi-level fusion and process decoupling, so as to obtain a network model that can simultaneously detect multiple types of building categories and quantitative height changes. The method of the present invention solves the problem of building quantitative height change detection, and does not need to use the height data of two phases. The introduced image modality data can effectively improve the detection frequency and flexibility, reduce the cost of quantitative height change detection, and its process decoupled annotation method is conducive to the description of complex urban building changes, and has strong practicality.

[0139] Those skilled in the art should be aware that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A process-decoupled multi-modal building height change detection method, the specific steps are as follows: S1. Construct a multi-mode cross-dimensional building height change detection model, including: Input module, multi-stage twin convolutional network, multimodal Gaussian guided aggregation module, decoding module; S2, input digital terrain models and image data of different periods, regardless of the previous and the next, into the input module, perform semi-automatic annotation based on the dual-phase digital terrain model DSM difference map and the building segmentation map, realize process-decoupled building category and height change annotation, and obtain a building height change detection training data set; The building height change detection training data set includes: DSM data of the previous phase, image data of the next phase, and the category change annotation map GT of the two phases. class-pre With GT class-post , and the building height change annotation map GT height ; S3, inputting the front-phase digital terrain model and the back-phase image data into a multi-level twin convolutional network with shared parameters in parallel to obtain a multi-modal cross-modal feature map, namely, DSM height features and image features; S4, based on the multimodal cross-modal feature map obtained in step S3, inputting the multimodal Gaussian guided aggregation module to perform height map guided multimodal feature fusion to obtain the final multi-scale cross-modal fusion feature; Based on the DSM height features and image features of the corresponding scales obtained in step S3, they are respectively subjected to variational Gaussianization, and the KL divergence loss of the feature layer distribution parameters and the standard normal distribution is calculated. Then, random resampling is performed using the standard normal distribution, and then multimodal feature fusion guided by the height map is performed, that is, the multimodal fusion features are used to predict the height change, and the average bi-norm loss is calculated with the height change annotation map; for the multi-scale multimodal fusion features, the adjacent scales are first upsampled and fused, and then upsampled step by step to the maximum scale to obtain the final multi-scale cross-modal fusion features; S5, based on the final multi-scale cross-modal fusion features obtained in step S4, input them into the decoding module, use two groups of three-channel 1×1 convolutions and one group of single-channel 1×1 convolutions for decoding and output, and obtain a three-channel category change prediction map representing the categories of the before and after phase changes, and a single-channel height change prediction map representing the height change; S6. Calculate the cross entropy loss of the output three-channel category change prediction map and the category change annotation map, calculate the average two-norm loss of the output single-channel height change prediction map and the height change annotation map, calculate the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module, and use gradient back propagation to minimize the loss, and correct the network parameters of the multimodal cross-dimensional building height change detection model; S7, training a multi-modal cross-dimensional building height change detection model, i.e., looping steps S3-S6, traversing all samples in the building height change detection training data set until the loss curve is smooth, and finally obtaining a trained building height change detection model that can be used for multi-modal cross-dimensional data; S8. Based on the trained building height change detection model obtained in step S7, the digital ground model to be inspected and the corresponding image data are input, and a category change prediction map representing the construction and demolition of buildings and a corresponding quantitative height change prediction map are output to achieve cross-modal quantitative detection of height changes of multiple categories of buildings.

2. The process-decoupled multi-modal building height change detection method according to claim 1, characterized in that: The step S2 is specifically as follows: Changes are annotated for multi-mode cross-dimensional data pairs composed of digital terrain models (DSM) and image data at different times. That is, the annotation process uses the dual-phase DSM difference map and the building segmentation map to pre-annotate the change area, and combines the definition of building height changes to achieve semi-automatic annotation; First, the digital terrain model H post With H pre Take the difference, that is, H post -H pre Get the height difference map H diff ; Merge the building areas of the previous and next phases respectively to obtain the merged image S of the building areas of the two phases building-pre With S building-post ; Then the height difference map H diff Merge map with building area S building-pre With S building-post Intersect and obtain the preliminary building area height change map of the two phases, and use the threshold method to mark the building area height change map to obtain a preliminary building category change mark map; The threshold method classifies building changes with a height change ΔH greater than a set threshold h as newly built, classifies building changes with a height change ΔH less than 0 as demolished, and classifies building changes with a height change ΔH greater than or equal to 0 but less than or equal to the set threshold h as no change; Set the building category change annotation map to 0, 1, and 2 to represent no building, building, and new building, respectively. Then, encode the two phases according to the building area height change map of the two phases, that is, use two coding positions to describe a total of 6 types of ground feature coverage changes in the two phases, including: in areas with demolition changes, use 1-0 to indicate that there was a building in the previous phase, and use 1-2 to indicate that there was a building in both phases; in areas with new construction changes, use 0-1 to indicate that there was a building in the next phase, and use 1-2 to indicate that there was a building in both phases; use 1-1 to indicate a building area with no building changes; use 0-0 to indicate a ground area with no building changes; Finally, the preliminary building category change annotation map is manually adjusted, including: adjusting the discontinuous building outline edges and abnormal height changes caused by tree occlusion, to obtain the final building category change annotation map GT class-pre , perform XOR operation on the category change annotation graphs of the two phases, and then compare them with the building height change graph H diff Intersect and get the building height change annotation map GT height ; DSM data of the previous phase, image data of the next phase, and the category change annotation map of the two phases GT class-pre With GT class-post , and the building height change annotation map GT height , forming a building height change detection dataset.

3. The process-decoupled multi-modal building height change detection method according to claim 1, characterized in that: The step S3 is specifically as follows: In each training, the pre-phase DSM and post-phase image data are input into the multi-level twin convolutional network in parallel to obtain the multi-scale feature maps of the two modalities. The multi-modal features of any scale are derived from the features of the digital terrain model. and image feature F I i constitute; The multi-stage twin convolutional network includes Z-stage convolutional blocks connected in series, and the convolutional blocks are used stage by stage to process two different modal inputs; The number of convolution blocks connected in series in the multi-level twin convolutional network is set according to actual conditions; Image modality input Directly use Z-level concatenated convolution blocks to process and obtain multi-scale feature output Each convolution block consists of 3 to 5 serially connected 3×3 convolutional layers, batch normalization layers, and activation function layers. The output of the activation function layer is added to the input of the convolutional layer as a residual connection. A convolutional layer with a stride of 2 is used for downsampling before each level of input, and the scale of the next level in the Z-level output is 0.5 times that of the previous level. The DSM modality input passes through a Z-level series of convolution blocks that share the input parameters of the imaging modality to obtain a multi-scale feature output.

4. The process-decoupled multi-modal building height change detection method according to claim 1, characterized in that: The step S4 is specifically as follows: Based on step S3, the previous phase DSM and the next phase image data are input into the multi-level twin convolutional network in parallel to obtain a multimodal cross-modal feature map. as well as After that (i=1,2,3,…Z), for two different modal features of any scale, variational Gaussianization is performed, that is, the mean and variance of the features are obtained by using 1×1 convolution mapping, and the expressions are as follows: in, represents the mean and variance of the DSM height feature, Represents the mean and variance of the image features; then calculate the Kullback-Leibler divergence of the distribution corresponding to these two sets of means and variances and the standard normal distribution, that is, use the Kullback-Leibler divergence loss to constrain and The corresponding mean and variance satisfy the normal distribution, and the expressions are as follows: Among them, p1(x) represents or The distribution obeyed, p2(x) represents the expected approximation distribution; when p2(x) ~ N(0,1), the KL loss expression is as follows: Among them, KL(p1||p2) represents the KL divergence loss of calculating distributions p1 and p2, that is, measuring the degree of difference between one probability distribution and another probability distribution; and σ and μ are replaced by in actual operation or By minimizing the above KL loss, or The mean and variance satisfy the standard normal distribution; the distribution constraint loss expression of the i-th layer of multimodal features to be fused is defined as follows: Then use a standard normal distribution to resample the mean and variance after variational Gaussianization, and splice the resampled features according to the channel, that is, the two sets of means and variances obtained by the constraints Use another standard normal distribution ε to sample and add it to the original features before Gaussianization. The expression is as follows: Among them, ε~N(0,1), and represents the final variational Gaussian modal enhancement feature; the enhanced features of the two modalities are concatenated by channel to obtain the modal fusion feature at this scale; Fusion feature F i The building height change annotation map is used to guide aggregation as the cross-modal feature of the current scale level. The expression is as follows: in, Indicates channel-by-channel concatenation. Tanh maps the input to the range of [-1,1]. F i (i,j) represents the i-th row and j-th column of the joint feature, GT height (i,j) represents the i-th row and j-th column on the real height change graph. represents the bi-norm loss between the height change prediction map and the height change annotation map output by the i-th level multimodal fusion feature; M represents the total number of rows of the annotation map, and N represents the total number of columns of the annotation map; The length and width of the next-level feature map are 0.5 times that of the previous-level feature map. For cross-modal features of multiple scales, adjacent scales are first upsampled and fused, and then upsampled to the maximum scale step by step. That is, 2 times upsampling is used to add and fuse adjacent multi-scale feature maps to obtain a multi-scale feature map group. The expression is as follows: Among them, UP 2× Indicates 2x upsampling; Then all scale features are upsampled to the first-level feature scale, and multi-scale feature maps are concatenated to obtain the final multi-scale cross-modal fusion feature F out , the expression is as follows:

5. The process-decoupled multi-modal building height change detection method according to claim 1, characterized in that: The step S5 is specifically as follows: The final multi-scale cross-modal fusion feature F obtained in step S4 out , using two groups of three-channel 1×1 convolutions and one group of single-channel 1×1 convolutions, activated by softmax and tanh functions respectively, we get the category change prediction map and the height change prediction map, the expressions are as follows: Pred class-pre =softmax(con1×1(F out )) Pred class-post =softmax(con1×1(F out )) Pred height =Tanh(con1×1(F out )) Among them, the softmax function normalizes each point of the three-channel category change prediction map to the range of 0 to 1 according to the channel dimension. The first channel represents the probability of no building, the second channel represents the probability of having a building, and the third channel represents the probability of building a new building; the Tanh function normalizes the single channel output to the range of -1 to 1; Pred class-pre Indicates the prediction result of the previous phase, Pred class-post Indicates the prediction result of the next phase, Pred height Represents the height change prediction map.

6. The process-decoupled multi-modal building height change detection method according to claim 1, characterized in that: The step S6 is specifically as follows: First, the cross entropy loss of the output three-channel category prediction map and the category change annotation map is calculated, the average two-norm loss of the single-channel height change prediction map and the height change annotation is calculated, and the KL divergence and average two-norm loss in the multimodal Gaussian guided aggregation module are calculated. The weighted sum of the above losses is the final training loss. The training loss is minimized using gradient back propagation to correct the network parameters of the multimodal cross-dimensional building height change detection model. The cross entropy loss function is used as the cost function, and the calculation expression is as follows: Among them, GT class-pre (i,j) and GT class-post (i,j) represents the i-th row and j-th column on the two time-phase category annotation maps, with values ​​of 0, 1, and 2, representing no buildings, buildings, and new buildings, respectively; M represents the total number of rows in the annotation map, and N represents the total number of columns in the annotation map; The two-norm loss is used as the height change cost function, and the calculation expression is as follows: The final training cost function is loss class-pre 、loss class-post 、loss height The weighted sum of is expressed as follows: Among them, λ1, λ2, λ3, and λ4 are constants greater than 0, representing the weight of each loss, which can be fine-tuned according to the actual application data set; by minimizing the loss, the final change detection model is obtained.

Images