Building intelligent monitoring method based on cross-modal multi-level

Abstract

The invention discloses an intelligent building monitoring method based on cross-modal multi-level, which takes a high-resolution remote sensing image and existing building vector data as sources, encodes correlation by fusing cross-modal information by using a collaborative attention module, realizes multi-level information extraction from top to bottom based on a pyramid structure, improves the sensing ability, and improves the accuracy of building monitoring. And building roof positioning and change evaluation are carried out through displacement and segmentation prediction. According to the intelligent building monitoring method based on cross-modal and multi-level, challenges related to large-scale data storage and retrieval are relieved through a cross-modal and multi-level building intelligent monitoring mode aiming at the problem of relatively high resource consumption in dual / multi-temporal remote sensing image building monitoring; meanwhile, prior information is fused based on multiple levels, influences of radiation differences, complex and diversified ground features and projection differences are avoided, and the building change detection precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of building change detection, and in particular to a building intelligent monitoring method based on cross-modal multi-level. Background Art

[0002] With the significant improvement of satellite imaging technology and resolution, as well as the rapid development of remote sensing deep learning, the use of high-resolution dual-phase satellite images for deep learning-based building change detection has become the preferred strategy. However, these methods usually only use single-modality image data, ignoring the prior information such as the specific geometric structure of the building, and do not take into account the image point displacement caused by terrain undulations and the difference in building position between images caused by different projection differences, resulting in extremely high false alarms. At the same time, the dual-phase image monitoring method requires a large amount of storage resources to save images. As the amount of data increases, the monitoring process becomes complicated, and due to factors such as radiation conditions and complex and diverse ground coverage, it also faces problems such as difficulty in detection and incompleteness. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a cross-modal multi-level intelligent building monitoring method, which aims to provide a method that reduces dependence on remote sensing images, alleviates the challenges associated with large-scale data storage and retrieval, avoids the influence of radiation differences, complex and diverse ground objects, and projection differences in dual-phase image monitoring, ensures the stability and reliability of the results, and provides a more cost-effective and interpretable option for long-term monitoring.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] A cross-modal multi-level intelligent building monitoring method includes the following steps:

[0006] S1: Extract features from the input original image and building polygons respectively, and use cross-modal building correlation mining to solve the change detection problem with target offset using the cross-correlation attention mechanism;

[0007] S2: Connect the features, based on the fusion of deep and shallow features, realize top-down multi-level information extraction based on the pyramid structure to improve perception ability;

[0008] S3: Design displacement and segmentation objective functions to locate and evaluate the changes of building roofs respectively, and integrate them to complete the monitoring.

[0009] Furthermore, in step S1, the cross-modal building correlation mining includes the following steps: based on the feature extraction of the classic network-VGG16, the input original image and the building polygon are fused with cross-modal information using a collaborative attention module to encode the correlation, thereby obtaining corresponding enhanced features;

[0010] For the two input features F a and F b , CC first calculates the relationship matrix S∈R (h×w)×(h×w) , the formula is as follows:

[0011]

[0012] W=P -1 DP

[0013] Among them, F a ∈R C×(h×w) and F b ∈R C×(h×w) is the feature extracted by the feature extraction network from the two images, h and w represent the height and width of the feature respectively, W∈R C×C is the weight matrix, P is a reversible matrix, D is a diagonal matrix, and the row and column dimensions of the features are normalized by the softmax function, namely:

[0014] S c =softmax(S),S r =softmax(S T )

[0015]

[0016] Among them, softmax(·) means normalizing the relevant features, S c ∈R h×w and S r ∈R h×w They are the column normalization results and row normalization results, and the column normalization result S c Indicates F a F b The correlation of the row normalization result S r Indicates F b F a relevance;

[0017] Then, the correlation features are used as attention weights to enhance the original features f a and f b :

[0018] Z a =F b S c ,Zb =F a S r

[0019] In order to weight the information between different images separately, that is, to process all the information of interest differently, a gated attention mechanism is introduced to obtain better weight distribution:

[0020] f g (Z a )=σ(w f Z a +b f )∈[0,1] h×w

[0021] f g (Z b )=σ(w f Z b +b f )∈[0,1] h×w

[0022] Where σ is the sigmoid activation function, w f and b f Represent the convolution kernel and bias respectively, f g Determines the degree of information retention and automatic learning. At this time, the characteristics of the enhanced collaborative attention mechanism can be expressed as:

[0023] Z a =Z a ★f g (Z a ),Z b =Z b ★f g (Z b )

[0024] Among them, the * operator is the channel-by-channel Hadamard product, that is, multiplying the elements in each channel by position;

[0025] Finally, the concatenated and enhanced features are passed through the convolution block to calculate the area where the common target is located, and the feature f combined with the attention mechanism is obtained. ′ d :

[0026] f = concat(Z a ,Z b )

[0027] A(f)=σ(MLP(f))=σ(W 2 (W 1 (W 0 (f))))

[0028] f ′d =(1+A(f))×f d

[0029]

[0030] Where σ represents the sigmoid function, W 0 ∈R 1×1×C×C / r , W 1 ∈R 3×3×C / r×C / r and W 2 ∈R 1×1×C / r×C / r They represent the weights of the convolution kernel sizes of 1*1, 3*3 and 1*1 respectively, r represents the channel reduction ratio, and f d represents the features after connection.

[0031] Furthermore, in step S2, the top-down multi-level feature fusion includes the following steps: using the fusion method of deep features and shallow features, the features extracted by the change detection network have a total of 4 layers, and the first three layers of features are regarded as shallow features: l {f 1 ,f 2 ,f 3}, the top-level features after the cross-correlation attention module are deep features: f h {f 4}, f h and f l Channel enhancement is performed separately, and then the deep features f h The global information is obtained by global pooling and convolution, and is used to enhance the shallow information to obtain f' l , referring to the classic feature pyramid network, multi-scale features are fused in a "top-down" manner until the original resolution is restored. In the pyramid structure, except for the top-level features, the remaining features of each layer Both accept two inputs, one is the upper layer feature after upsampling, and the other is the connection feature of this layer After adding each element, the features of this layer are generated through 1*1 convolution. Perform the same process on each layer of the pyramid; finally concatenate the multi-level features Carry out information fusion.

[0032] Furthermore, in step S3, the building roof positioning and change assessment includes the following steps: after information fusion, the displacement and segmentation branches are designed to respectively perform building roof positioning and change assessment, and they are integrated to complete the monitoring; displacement loss Mainly used to calculate the mean square error between the predicted displacement and the ground truth displacement, segmentation loss It is to predict the corresponding category of each pixel, including background, polygon interior, boundary and corner point. The overall objective function is as follows:

[0033]

[0034] For the displacement loss function, since the displacement of each pixel on the image cannot be predicted equally well, the corner points of the building can be matched accurately, the boundary pixels will be blurred along one dimension, and the pixels inside the area will be ambiguous in two spatial dimensions, making it difficult to match the exact displacement; therefore, the input polygon pixels are divided into four categories: background, polygon interior, boundary, and corner points, and the displacement loss is calculated based on adding different weights to each category. The formula is as follows:

[0035]

[0036] Among them, H and W represent the height and width of the discrete image respectively, c(x)∈{1,2,3,4} represents the pixel category, namely background, polygon interior, boundary and corner point, and w c is the loss coefficient for each pixel class, n c is the total number of pixels of each class, and f gt (x) is the predicted displacement and the ground truth displacement;

[0037] In the segmentation loss function, the probability of each pixel belonging to the four categories of background, polygon interior, boundary and corner is predicted. And use class-specific cross entropy Perform calculations, is the true value of the pixel category, D(p) represents the distribution of two possibilities (p, 1-p), and also adds different weights w' to each type c , the formula is as follows:

[0038]

[0039] The beneficial effects of the present invention are:

[0040] The present invention provides a cross-modal multi-level intelligent building monitoring method, which aims to solve the problem of large resource consumption in dual / multi-temporal remote sensing image building monitoring, and alleviates the challenges related to large-scale data storage and retrieval through a cross-modal multi-level intelligent building monitoring method; at the same time, based on the multi-level integration of prior information, it avoids the influence of radiation differences, complex and diverse ground objects, and projection differences, and improves the accuracy of building change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the cross-modal multi-level intelligent monitoring method for buildings of the present invention;

[0042] Figure 2 A schematic diagram of cross-modal building correlation mining of the present invention;

[0043] Figure 3 A schematic diagram of feature extraction for a change detection network of the present invention;

[0044] Figure 4 It is a schematic diagram of multi-level feature fusion of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings, wherein the same components are represented by the same reference numerals.

[0046] It should be noted that the words "front", "rear", "left", "right", "up" and "down" used in the following description refer to directions in the drawings, and the words "inside" and "outside" refer to directions toward or away from the geometric center of a specific component, respectively.

[0047] In order to make the contents of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0048] like Figure 1 As shown in the figure, a cross-modal multi-level intelligent building monitoring method is proposed. It uses high-resolution remote sensing images and existing building vector data as input to explore and solve some technical difficulties faced by building monitoring in complex scenes. First, the features of the input original image and building polygon are extracted respectively, and the cross-modal building correlation mining is used to solve the change detection problem with target offset by using the cross-correlation attention mechanism; then, the features are connected, and on the basis of deep and shallow feature fusion, the top-down multi-level information extraction is realized based on the pyramid structure to improve the perception ability and ensure the stability and reliability of the results; finally, the displacement and segmentation objective functions are designed to locate and evaluate the building roof respectively, and they are integrated to complete the monitoring.

[0049] (1) Cross-modal building correlation mining

[0050] like Figure 2As shown in the figure, the displacement of image points caused by terrain undulation will cause projection difference on the image, that is, the target object conformation deviates from the correct position of its orthographic projection. For high-rise buildings, the roof position does not overlap with the base and is tilted. For building base vector data and high-resolution remote sensing images, they often cannot be completely matched, and there will always be a certain geometric offset, which makes it difficult for many monitoring methods to achieve ideal results. To this end, cross-modal building correlation mining is studied to constrain building monitoring. Based on the classic network-VGG16 feature extraction, the input original image and building polygon are fused with cross-modal information using a collaborative attention module to encode the correlation, thereby obtaining the corresponding enhanced features, so as to fully mine the relationship between buildings and provide a guarantee for reducing or even eliminating the impact of geometric displacement on building monitoring.

[0051] For the two input features F a and F b , CC first calculates the relationship matrix S∈R (h×w)×(h×w) , the formula is as follows:

[0052]

[0053] W=P -1 DP

[0054] Among them, F a ∈R C×(h×w) and F b ∈R C×(h×w) is the feature extracted by the feature extraction network from the two images, h and w represent the height and width of the feature respectively, W∈R C×C is the weight matrix, P is a reversible matrix, D is a diagonal matrix, and the row and column dimensions of the features are normalized by the softmax function, namely:

[0055] S c =softmax(S),S r =softmax(S T )

[0056]

[0057] Among them, softmax(·) means normalizing the relevant features, S c ∈R h×w and S r ∈R h×w They are the column normalization results and row normalization results, and the column normalization result S c Indicates F a F b The correlation of the row normalization result S r Indicates Fb F a of relevance.

[0058] Then, the correlation features are used as attention weights to enhance the original features f a and f b :

[0059] Z a =F b S c ,Z b =F a S r

[0060] In order to weight the information between different images separately, that is, to process all the information of interest differently, a gated attention mechanism is introduced to obtain better weight distribution:

[0061] f g (Z a )=σ(w f Z a +b f )∈[0,1] h×w

[0062] f g (Z b )=σ(w f Z b +b f )∈[0,1] h×w

[0063] Where σ is the sigmoid activation function, w f and b f Represent the convolution kernel and bias respectively, f g Determines the degree of information retention and automatic learning. At this time, the characteristics of the enhanced collaborative attention mechanism can be expressed as:

[0064] Z a =Z a ★f g (Z a ),Z b =Z b ★f g (Z b )

[0065] Among them, the ★ operator is the channel-by-channel Hadamard product, that is, multiplying the elements on each channel by position.

[0066] Finally, the concatenated and enhanced features are passed through the convolution block to calculate the area where the common target is located, and the feature f' combined with the attention mechanism is obtained d :

[0067] f = concat(Z a ,Z b )

[0068] A(f)=σ(MLP(f))=σ(W 2 (W 1 (W 0 (f))))

[0069] f' d =(1+A(f))×f d

[0070]

[0071] Where σ represents the sigmoid function, W 0 R 1×1×C×C / r、W 1 R 3×3×C / r×C / r and W 2 R 1×1×C / r×C / r They represent the weights of the convolution kernel sizes of 1*1, 3*3 and 1*1 respectively, r represents the channel reduction ratio, and f d represents the features after connection.

[0072] (2) Top-down multi-level feature fusion

[0073] Generally speaking, among the features extracted by deep neural networks, deep features have more semantic information and global information, but lose spatial structure information, while shallow networks retain spatial structure better but have no semantic clues. In order to make up for these defects and make features retain spatial information while having semantic information, a fusion method of deep features and shallow features is usually used. The features extracted by the change detection network proposed in the present invention have a total of 4 layers, and the first three layers of features are regarded as shallow features: l {f 1 ,f 2 ,f 3}, the top-level features after the cross-correlation attention module are deep features: f h {f 4}. f h and f l Channel enhancement is performed separately, and then the deep features f h The global information is obtained by global pooling and convolution, and is used to enhance the shallow information to obtain f' l ,like Figure 3 As shown. Referring to the classic feature pyramid network, multi-scale features are fused in a "top-down" manner until the original resolution is restored. In order to make full use of multi-scale information, the present invention uses the pyramid structure to store all features except the top-level features. Both accept two inputs, one is the upper layer feature after upsampling, and the other is the connection feature of this layer After adding each element, the features of this layer are generated through 1*1 convolution. Perform the same process on each layer of the pyramid; finally concatenate the multi-level features To perform information fusion, such as Figure 4 shown.

[0074] (3) Building roof positioning and change assessment

[0075] After information fusion, the displacement and segmentation branches are designed to locate and evaluate the rooftop changes of the building respectively, and then they are integrated to complete the monitoring; displacement loss Mainly used to calculate the mean square error between the predicted displacement and the ground truth displacement, segmentation loss It is to predict the corresponding category of each pixel, including background, polygon interior, boundary and corner point. The overall objective function is as follows:

[0076]

[0077] For the displacement loss function, since the displacement of each pixel on the image cannot be predicted equally well, the corner points of the building can be matched accurately, the boundary pixels will be blurred along one dimension, and the pixels inside the area will be ambiguous in two spatial dimensions, making it difficult to match the exact displacement; therefore, the input polygon pixels are divided into four categories: background, polygon interior, boundary, and corner points, and the displacement loss is calculated based on adding different weights to each category. The formula is as follows:

[0078]

[0079] Among them, H and W represent the height and width of the discrete image respectively, c(x)∈{1,2,3,4} represents the pixel category, namely background, polygon interior, boundary and corner point, and w c is the loss coefficient for each pixel class, n c is the total number of pixels of each class, and f gt (x) is the predicted displacement and the ground truth displacement;

[0080] In the segmentation loss function, the probability of each pixel belonging to the four categories of background, polygon interior, boundary and corner is predicted. And use class-specific cross entropy Perform calculations, is the true value of the pixel category, D(p) represents the distribution of two possibilities (p, 1-p), and also adds different weights w' to each type c , the formula is as follows:

[0081]

[0082] The present invention uses high-resolution remote sensing images and existing building vector data as sources, and utilizes a collaborative attention module to fuse cross-modal information to encode correlations, thereby obtaining corresponding enhanced features and fully exploring the relationships between buildings. Based on the fusion of deep and shallow features, a pyramid structure is used to extract multi-level information from top to bottom to improve perception capabilities and thereby ensure the stability and reliability of the results. Displacement and segmentation objective functions are designed to respectively locate and evaluate building rooftop changes, which are then integrated to complete monitoring.

[0083] The present invention provides a cross-modal multi-level intelligent building monitoring method, which aims to solve the problem of large resource consumption in dual / multi-temporal remote sensing image building monitoring, and alleviates the challenges related to large-scale data storage and retrieval through a cross-modal multi-level intelligent building monitoring method; at the same time, based on the multi-level integration of prior information, it avoids the influence of radiation differences, complex and diverse ground objects, and projection differences, and improves the accuracy of building change detection.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal multi-level intelligent building monitoring method, characterized in that: The following steps are involved: S1: Extract features from the input original image and building polygons respectively, and use cross-modal building correlation mining to solve the change detection problem with target offset using the cross-correlation attention mechanism; S2: Connect the features, based on the fusion of deep and shallow features, realize top-down multi-level information extraction based on the pyramid structure to improve perception ability; S3: Design displacement and segmentation objective functions to locate and evaluate the changes of building roofs respectively, and integrate them to complete the monitoring.

2. The method for intelligent building monitoring based on cross-modal multi-level according to claim 1, characterized in that: In step S1, the cross-modal building correlation mining includes the following steps: based on the classic network-VGG16 feature extraction, the input original image and building polygon are fused with cross-modal information using a collaborative attention module to encode the correlation, thereby obtaining corresponding enhanced features; For the two input features F a and F b , CC first calculates the relationship matrix S∈R (h×w)×(h×w) , the formula is as follows: W=P -1 DP Among them, F a ∈R C×(h×w) and F b ∈R C×(h×w) is the feature extracted by the feature extraction network from the two images, h and w represent the height and width of the feature respectively, W∈R C×C is the weight matrix, P is a reversible matrix, D is a diagonal matrix, and the row and column dimensions of the features are normalized by the softmax function, namely: S c =softmax(S),S r =softmax(S T ) Among them, softmax(·) means normalizing the relevant features, S c ∈R h×w and S r ∈R h×w They are the column normalization results and row normalization results, and the column normalization result S c Indicates F a F b The correlation of the row normalization result S r Indicates F b F a relevance; Then, the correlation features are used as attention weights to enhance the original features f a and f b : Z a =F b S c ,Z b =F a S r In order to weight the information between different images separately, that is, to process all the information of interest differently, a gated attention mechanism is introduced to obtain better weight distribution: f g (WITH a )=σ(in f WITH a +b f )∈[0,1] h×w f g (WITH b )=σ(in f WITH b +b f )∈[0,1] h×w Where σ is the sigmoid activation function, w f and b f Represent the convolution kernel and bias respectively, f g Determines the degree of information retention and automatic learning. At this time, the characteristics of the enhanced collaborative attention mechanism can be expressed as: WITH a =Z a *f g (WITH a ),WITH b =Z b *f g (WITH b ) Among them, the ★ operator is the channel-by-channel Hadamard product, that is, multiplying the elements in each channel by position; Finally, the concatenated and enhanced features are passed through the convolution block to calculate the area where the common target is located, and the feature f combined with the attention mechanism is obtained. ′ d : f=concat(Z a ,Z b ) A(f)=σ(MLP(f))=σ(W2(W1(W0(f)))) f ′ d =(1+A(f))×f d Where σ represents the sigmoid function, W0∈R 1×1×C×C / r ,W1∈R 3×3×C / r×C / r and W2∈R 1×1×C / r×C / r They represent the weights of the convolution kernel sizes of 1*1, 3*3 and 1*1 respectively, r represents the channel reduction ratio, and f d represents the features after connection.

3. The method for intelligent building monitoring based on cross-modal multi-level according to claim 1, characterized in that: In step S2, the top-down multi-level feature fusion includes the following steps: using the fusion method of deep features and shallow features, the features extracted by the change detection network have a total of 4 layers, and the first three layers of features are regarded as shallow features: l {f1,f2,f3}, the top-level features after the cross-correlation attention module are deep features: f h {f4}, f h and f l Channel enhancement is performed separately, and then the deep features f h The global information is obtained by global pooling and convolution, and is used to enhance the shallow information to obtain f' l , referring to the classic feature pyramid network, multi-scale features are fused in a "top-down" manner until the original resolution is restored. In the pyramid structure, except for the top-level features, the remaining features of each layer Both accept two inputs, one is the upper layer feature after upsampling, and the other is the connection feature of this layer After adding each element, the features of this layer are generated through 1*1 convolution. Perform the same process on each layer of the pyramid; finally, concatenate the multi-level features Carry out information fusion.

4. The method for intelligent building monitoring based on cross-modal multi-level according to claim 1, characterized in that: In step S3, the building roof positioning and change assessment includes the following steps: after information fusion, the displacement and segmentation branches are designed to respectively perform building roof positioning and change assessment, and then integrate them to complete the monitoring; displacement loss Mainly used to calculate the mean square error between the predicted displacement and the ground truth displacement, segmentation loss It is to predict the corresponding category of each pixel, including background, polygon interior, boundary and corner point. The overall objective function is as follows: For the displacement loss function, since the displacement of each pixel on the image cannot be predicted equally well, the corner points of the building can be matched accurately, the boundary pixels will be blurred along one dimension, and the pixels inside the area will be ambiguous in two spatial dimensions, making it difficult to match the exact displacement; therefore, the input polygon pixels are divided into four categories: background, polygon interior, boundary, and corner points, and the displacement loss is calculated based on adding different weights to each category. The formula is as follows: Among them, H and W represent the height and width of the discrete image respectively, c(x)∈{1,2,3,4} represents the pixel category, namely background, polygon interior, boundary and corner point, and w c is the loss coefficient for each pixel class, n c is the total number of pixels of each class, and f gt (x) is the predicted displacement and the ground truth displacement; In the segmentation loss function, the probability of each pixel belonging to the four categories of background, polygon interior, boundary and corner is predicted. And use class-specific cross entropy Perform calculations, is the true value of the pixel category, D(p) represents the distribution of two possibilities (p, 1-p), and also adds different weights w' to each type c , the formula is as follows:

Images