A multispectral image change detection method based on multi-scale adaptive convolution kernel
By introducing multi-scale adaptive convolution kernel and attention mechanism in multi-spectral image change detection, combined with multi-modal conditional random field, the problem of inaccurate extraction of complex geographic information and boundary positioning is solved, and more accurate changes are achieved Test results.
Patent Information
- Application Number
- CN202210474822.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-04-29
AI Technical Summary
When the existing multispectral image change detection method processes complex geographic information, it is difficult to extract multi-scale features, resulting in inaccurate positioning of the boundary of the changing region.
Using a multi-spectral image change detection method based on multi-scale adaptive convolution kernel, multi-scale features are extracted by introducing the adaptive convolution kernel selection module SCKB and attention mechanism in the network MASK-Net with a two-sided encoding path. Multimodal condition random field is used to perform secondary detection to refine the boundaries.
The positioning accuracy of the boundaries of the changing region is improved and the feature expression ability of complex landform information is enhanced. The experimental results perform better than other methods on multiple data sets, verifying the effectiveness of the method.
Smart Images

Figure CN114841319B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a multispectral image change detection method and belongs to the field of multispectral image change detection. Background Art
[0002] Multispectral image change detection is the detection of change information in multispectral images of two or more time images of the same area. Due to changes in human activities and ecosystems, the collection and analysis of surface change information is of great significance to the study of environmental protection, natural resource management, and the relationship between human social development and the natural environment.
[0003] With the development of remote sensing technology, the spatial resolution of multispectral images has been continuously improved, and the rich ground object details make up for the lack of spectral resolution. Multispectral images have more advantages in detecting some obvious change types of ground objects (such as building demolition, road changes, etc.). However, using a single-scale convolution kernel to extract change features is not suitable for the complex ground object information of multispectral images. Therefore, it is considered to introduce multi-scale information to improve the ability to extract complex ground object information. Deep neural network (DNN) is more suitable for processing detection tasks in such complex scenes. Many scholars use neural networks to solve the problems in multispectral change detection. Since the U-Net network model was proposed, it has been widely used in multiple image segmentation fields such as remote sensing image segmentation and medical image segmentation. With the application of neural networks, multispectral change detection has gradually transitioned to an object-level detection method. Relying solely on neural networks to distinguish the change area can easily lead to inaccurate positioning of the detected change area boundary. Subsequent processing technology is required to solve small-scale misclassification and refine the classification boundary. Therefore, how to extract multi-scale features for complex ground object information and perform secondary classification to refine the boundary information of the object is a key and urgent problem to be solved. Summary of the invention
[0004] In order to solve the problems in the process of hyperspectral image change detection that the existing methods have difficulty in extracting complex ground object information and inaccurately locating the boundary of the changed area, the present invention proposes a multispectral image change detection method based on a multi-scale adaptive convolution kernel.
[0005] A multispectral image change detection method based on multi-scale adaptive convolution kernels is proposed. The multispectral image is input into the weight-sharing bilateral encoding path network MASK-Net for detection.
[0006] The network MASK-Net of the bilateral encoding path simultaneously extracts independent features of the dual-phase multispectral image; each encoding path includes four layers of convolutional networks connected in sequence, and a maximum pooling layer is set between every two layers of the four-layer convolutional network; the first two layers of the convolutional network in the four-layer convolutional network each include two convolutional layers with 3×3 convolutional kernels and one BN layer; the last two layers of the convolutional network each include two SCKB modules and one BN layer;
[0007] Then, the shallow features extracted by the encoding path are input into the decoding path to extract deep features. The decoding path includes four upsampling modules. The input of the first upsampling module is formed by superimposing the encoding results of the two encoding paths. The input features of the following three upsampling modules are directly superimposed by the output features of the previous upsampling module and the output features of the two encoding paths of the same level. The upsampling module with an attention mechanism is used to extract the deep features in the network, and selectively fuses them with the shallow features extracted by the encoding path. After passing through four upsampling modules, the output features of the last upsampling module are passed through a convolution layer with a 1×1 convolution kernel to adjust the number of channels of the final change detection map, and output the change detection result.
[0008] The SCKB module is a convolution kernel selection module based on adaptive weights; the SCKB module is divided into three convolution branches, each of which includes a convolution layer, a BN layer, and an activation layer. The convolution kernel sizes in the convolution layers of the three convolution branches are 3×3, 5×5, and 7×7, respectively, which are used to extract features U1, U2, and U3 of three sizes; the features U1, U2, and U3 based on the three sizes are superimposed on the channel dimension to obtain a multi-scale feature map U:
[0009] U=[U1;U2;U3]=[Conv 3×3 (F1); Conv 5×5 (F1); Conv 7×7 (F1)]
[0010] Among them, Conv() represents the convolution operation, F1 is the input of the SCKB module; [;] in [U1; U2; U3] means that these three feature maps are superimposed on the channel dimension through the concat operation;
[0011] Then, global information is encoded through global pooling to generate a feature vector S in the channel dimension. The first third of the feature vector S represents the statistical information corresponding to the 3×3 convolution kernel, the second third represents the statistical information corresponding to the 7×7 convolution kernel, and the middle represents the statistical information corresponding to the 5×5 convolution kernel.
[0012] Then, two one-dimensional convolutions are introduced to fuse all statistical information, and the feature vector output by the second one-dimensional convolution is reshaped into a score matrix; the score matrix is mapped to the weight coefficient matrix of the three convolution kernels through softmax calculation, and each column in the weight coefficient matrix corresponds to the weight of the three-size features U1, U2, and U3 output by the three convolution kernels at the channel; finally, the weighted values of U1, U2, and U3 and the corresponding weights are calculated to obtain multi-scale fusion features; then, after a BN layer, the maximum pooling layer is used to obtain high-dimensional features.
[0013] Furthermore, the cth element of the feature vector S is calculated as follows:
[0014]
[0015] In the formula, h is the image height, w is the image width, m and n are the variables in the summation, the value range of m is (1, h), the value range of n is (1, w), and U c It is the feature map of the cth channel in the multi-scale feature U.
[0016] Furthermore, the convolutional layers in the first two convolutional networks of the encoding path are convolutional layers with activation functions, and the activation function is Relu.
[0017] Furthermore, the upsampling module is the UM module. The UM module first performs maximum pooling and average pooling on the input feature F2 in the spatial dimension to obtain two feature vectors with the same length as the number of F2 channels, and respectively sends the two feature vectors to two multi-layer perceptrons MLP, and then adds and fuses the two output results at the pixel level; then, based on the Sigmoid activation function, the addition result of the multi-layer perceptron is mapped to a weight vector between zero and one, that is, the weight vector M of the channel attention. c ;
[0018] The weight vector and input features are reorganized, and the optimization method of the channel attention reorganization feature is as follows:
[0019]
[0020] in, represents the optimized output features, Represents tensor product, that is, the multiplication of elements at corresponding positions of two tensors;
[0021] Then it is sent to two consecutive 3×3 convolutional layers to obtain feature F4;
[0022] The spatial attention map is then adaptively learned by the neural network layer:
[0023] M s=σ(Conv2D([AvgPool(F4);MaxPool(F4)]))
[0024] Where M s represents the spatial attention map; Conv2D represents the two-dimensional convolution; σ represents the Sigmoid activation function;
[0025] After obtaining the spatial attention map, the spatial reorganization feature is then calculated; the calculation method of the spatial reorganization feature is as follows:
[0026]
[0027] Where M s represents the spatial attention map, Represents tensor product, which is the multiplication of elements at corresponding positions of two tensors.
[0028] Furthermore, the MASK-Net is a trained MASK-Net, and the MASK-Net uses weighted cross entropy as the loss function during training:
[0029]
[0030] Where N is the total number of samples, t i Indicates the label of the i-th pixel. When the pixel belongs to the change area, t i is 1, otherwise, the pixel is a non-changing pixel. i is 0; i Represents the prediction result of the neural network for the i-th pixel, y i The larger the value, the more likely the neural network thinks that the pixel belongs to the change area. i represents the probability of the neural network predicting a non-changing pixel; w c Represents the category weight.
[0031] Furthermore, the multispectral image input into the network MASK-Net of the weight-shared bilateral encoding path is obtained by segmenting the entire acquired multispectral image.
[0032] Furthermore, the training set used to train MASK-Net needs to be processed with data enhancement. During the enhancement, the samples are selectively enhanced. If the proportion of changed pixels in the training sample is less than 1%, it is filtered so that it does not participate in the network training; if the proportion of changed pixels in the training sample is more than 5%, image enhancement is used for it, and each training sample is rotated and flipped 90°, 180°, and 270°; then the training sample is sent to the network for training.
[0033] Furthermore, the multispectral image change detection method based on a multi-scale adaptive convolution kernel also includes a step of performing secondary detection using a multimodal conditional random field. The specific process includes the following steps:
[0034] The multispectral image is input into MASK-Net for detection to obtain the prediction result of the network, and the unary potential function of the FC-CRF model is constructed with the prediction result; at the same time, the CVA difference map and the SA spectrum difference map are calculated for the multispectral image, and the two difference maps are used to construct the binary potential function of the FC-CRF model; and then a secondary detection is performed through the fully connected conditional random field;
[0035] Fully connected conditional random field is FC-CRF. The probability distribution of the FC-CRF output result Y is defined by the following formula:
[0036]
[0037] Where X represents the input, p1 and q1 are intermediate variables for summation, ranging from 1 to N, and φ u represents the unary potential function, φ p represents the binary potential function;
[0038] φ u The Softmax classification result output by the neural network is calculated using the formula:
[0039]
[0040] It indicates the probability strength that the neural network believes that pixel p2 belongs to the changed pixel;
[0041] The binary potential function is expressed as:
[0042]
[0043] in, and are binary potential functions defined based on the grayscale difference map extracted by the change vector analysis CVA and the spectral difference map calculated by the spectral angle SA, respectively. and are the weights of the two potential functions, p2 and q2 are the intermediate variables for summation;
[0044] The formula is as follows:
[0045]
[0046] in, is the label function, l is the summed intermediate variable, and K represents the number of Gaussian kernels;
[0047] The Gaussian kernel is as follows:
[0048]
[0049] in Respectively represent the position vectors of pixels p2 and q2, Respectively represent the difference intensity values of pixels p2 and q2 in the CVA difference map; σ α With σ β is the Gaussian kernel parameter in the previous Gaussian kernel; σ γ is the Gaussian kernel parameter in the previous Gaussian kernel; w1 and w2 are the Gaussian kernel weights;
[0050] For the SA spectrum difference map, the processing process is the same as same;
[0051] Finally, the mean field approximation algorithm is used to iteratively optimize the energy potential function of the FC-CRF model to derive the category label corresponding to each pixel.
[0052] Furthermore, when constructing the binary potential function, the calculation formula for the difference intensity of the CVA difference map at this point is:
[0053]
[0054] In the formula, i1 is the intermediate variable for summation;
[0055] For point p2, CVA(I x ,I y )The formula at pixel p2 is as follows:
[0056]
[0057] In the formula, It is the difference intensity value in the CVA difference map, that is
[0058] Let V x =(v x1 , v x2 , …, v xn ) and V y =(v y1 , v y2 , …, v yn ) is the spectral vector of the pixels in the same position of the two images, then SA(V x ,V y ) is as follows:
[0059]
[0060] Among them, i2 is the summation intermediate variable; SA(Vx ,V y ) ranges from 0 to π. The smaller the value of the spectral angle, the higher the similarity between the two feature vectors, and the greater the possibility that they belong to the same type of features. The calculation process and Similarly, in order to distinguish the physical meaning of the parameters in the calculation, In Denoted as It is used to represent the angular difference of the spectral vectors of pixels p2 and q2 in the SA difference map;
[0061] For point p2, It is the angle difference of the spectral vector in the SA difference map,
[0062] Furthermore, the label function As follows:
[0063]
[0064] Among them, w (l) is the Gaussian kernel The weight coefficient of and It is the feature vector corresponding to pixel points p2 and q2.
[0065] Beneficial effects:
[0066] The present invention proposes a multi-spectral change detection method based on multi-scale adaptive convolution kernel. Combined with the characteristics of multi-spectral data sources, a convolution kernel selection module (SCKB) that can adaptively allocate weights is proposed to address the problem that the current multi-spectral change detection method using a single convolution kernel does not fully utilize multi-scale information. Since the U-Net network framework uses jump connections to retain more original image information, the present invention uses U-Net with dual encoding paths as the overall framework. The above-mentioned convolution kernel selection module is embedded in the encoding path to extract multi-scale features in the image. At the same time, in order to overcome the problem of feature heterogeneity fusion, an attention mechanism is added to the decoding path to selectively fuse shallow features and deep features. Finally, a multimodal conditional random field is used to perform secondary detection on the detection results of the neural network, restore the local information lost by the neural network, and make the final detection boundary more accurate. The experimental results of the five change detection methods on two data sets are observed. The comprehensive indicators of this method on four different test sets are the highest, which verifies the effectiveness of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 This is a flowchart of multi-spectral change detection based on multi-scale adaptive convolution kernels;
[0068] Figure 2 This is a network framework diagram of the method proposed in the present invention;
[0069] Figure 3 This is the structure diagram of the adaptive weight-based convolution kernel selection module (SCKB) in step 3;
[0070] Figure 4 Schematic diagram of the upsampling module (UM) with attention mechanism in step 4;
[0071] Figure 5 The overall structure diagram of the multi-spectral change detection method based on multi-scale adaptive convolution kernel;
[0072] Figure 6-Figure 9 The detection results of the present invention on two multispectral data sets are: Figure 6-Figure 9 The first column is the image before the change, the second column is the image after the change, the third column is the true value map of the change, and the fourth column is the detection result of the proposed method. DETAILED DESCRIPTION
[0073] Specific implementation method 1: Combination Figure 1 To explain this embodiment,
[0074] The present embodiment provides a method for detecting changes in multispectral images based on a multi-scale adaptive convolution kernel, comprising the following steps:
[0075] Step 1: Select the image block size of the training data, divide the entire multispectral image into training data according to the image block size, and then select a part of the training samples after sample balancing and data enhancement, and input them into the multi-scale feature extraction network.
[0076] Step 2: The present invention expands the coding path of the U-Net network, designs a weight-shared bilateral coding path network MASK-Net, uses a supervised training method to retain the original features of the dual-phase multispectral image, and extracts the independent features of the dual-phase multispectral image at the same time without introducing additional parameters. Compared with the original U-Net network, this method can make full use of the spatial features of different scales in the multispectral image to extract the complex ground object information of the multispectral image.
[0077] Step 3: The present invention proposes a convolution kernel selection module (selective convolution kernel block, SCKB) based on adaptive weights by drawing on the idea of the attention mechanism and embeds it into the encoding path to extract multi-scale features from the image. The SCKB module is divided into three convolution branches. The weights assigned to each convolution kernel branch are calculated by drawing on the idea of the attention mechanism. Finally, the weighted values of each feature map and the corresponding weights are calculated to obtain the multi-scale fusion features. The SCKB module will automatically adjust the weights assigned to the three multi-scale features according to different application scenarios, allowing the network to select the most appropriate convolution kernel size by itself.
[0078] Step 4: The present invention embeds the designed upsampling module (UM) with attention mechanism into the decoding path to extract deep features in the U-Net network. Channel attention enables the selective fusion of shallow features and deep features in the U-Net results, while spatial attention suppresses the feature information of non-changing pixels and enhances the difference features of changing pixels. After the optimization of channel attention and spatial attention, the feature map has better expression of change information in both channel dimension and spatial dimension. The decoding path is composed of four upsampling modules UM. The present invention retains the jump connection of the U-Net network, that is, the input of each UM module is the superposition of the shallow features extracted by the corresponding encoding path of this layer and the deep features extracted by the UM module of the previous layer. The output of the last UM module passes through a convolution layer with a 1×1 convolution kernel to adjust the number of channels of the final change detection map, thereby obtaining a change detection result map.
[0079] Step 5: The multi-spectral change detection method based on multi-scale adaptive convolution kernel can already realize the classification and positioning of changed pixels, but there is still the problem of inaccurate positioning due to information loss. To address this problem, the present invention uses multimodal conditional random fields to perform secondary detection on the results of the neural network.
[0080] More specifically,
[0081] The specific process of step 1 includes the following steps:
[0082] One third of the test images in the two datasets were cut as validation sets, and the remaining two thirds were used as test sets. Data images other than test images were cropped into 128×128 image blocks for training sets.
[0083] The training set is enhanced. When enhancing, the samples need to be selectively enhanced. If the proportion of changed pixels in the training sample is less than 1%, it will be filtered so that it does not participate in network training; if the proportion of changed pixels in the training sample is more than 5%, image enhancement is used for it, and each training sample is rotated and flipped 90°, 180°, and 270°. In this way, the distribution of positive and negative sample data in the training sample is more balanced, avoiding too small variance. The training sample is then sent to the multi-scale feature extraction network for training.
[0084] The specific process of step 2 includes the following steps:
[0085] The encoding path of the U-Net network is designed based on the weight-sharing bilateral encoding path network MASK-Net, such as Figure 2 As shown in the figure, the original features of the dual-phase multispectral image are retained by supervised training, and the independent features of the dual-phase multispectral image are extracted without introducing additional parameters. Each encoding path is composed of four convolutional networks connected in sequence. The first two convolutional networks include two convolutional layers with 3×3 convolution kernels and a BN layer. The convolutional layers have activation functions, and the activation function uses Relu to alleviate the gradient disappearance. The last two convolutional networks include two SCKB modules and a BN layer; a 2×2 maximum pooling layer is set between every two layers of the four-layer convolutional network;
[0086] The entire network uses weighted cross entropy as the loss function, and its calculation formula is as follows:
[0087]
[0088] Where N is the total number of samples, t i Indicates the label of the i-th pixel. When the pixel belongs to the change area, t i is 1, otherwise, the pixel is a non-changing pixel. i is 0; i Represents the prediction result of the neural network for the i-th pixel, y i The larger the value, the more likely the neural network thinks that the pixel belongs to the change area. i represents the probability of the neural network predicting a non-changing pixel; w c Represents the category weight, which is generally the ratio of the number of non-changing samples to the number of changing samples, and is usually a value greater than 1.
[0089] The specific process of step three includes the following steps:
[0090] The present invention draws on the idea of attention mechanism to propose a convolution kernel selection module SCKB based on adaptive weights and embeds it into the encoding path to extract multi-scale features in the image, such as Figure 3 As shown. The SCKB module is divided into three convolution branches, each of which includes a convolution layer, a BN layer, and an activation layer (not an activation function set in the convolution layer, but a separately set activation layer). The convolution kernel sizes in the convolution layers of the three convolution branches are 3×3, 5×5, and 7×7, respectively, corresponding to different receptive fields, which are used to extract features of three sizes U1, U2, and U3. The features U1, U2, and U3 based on the three sizes are superimposed on the channel dimension to obtain a multi-scale feature map U:
[0091] U=[U1;U2;U3]=[Conv 3×3 (F1); Conv 5×5 (F1); Conv 7×7 (F1)]
[0092] Among them, Conv() represents the convolution operation, and F1 is the input of the SCKB module;
[0093] Assuming that the size of the input feature map F1 is (w, h, c), the size of the deep features obtained by the three convolution branches remains unchanged. The [;] in [U1; U2; U3] means that these three feature maps are superimposed on the channel dimension through the concat operation to obtain a multi-scale feature U with an integrated size of (w, h, 3c).
[0094] Then, global information is encoded through global pooling to generate a feature vector S in the channel dimension. The first third of S represents the statistical information corresponding to the 3×3 convolution kernel, the second third represents the statistical information corresponding to the 7×7 convolution kernel, and the middle represents the statistical information corresponding to the 5×5 convolution kernel. The cth element of the feature vector S is calculated as follows:
[0095]
[0096] In the formula, h is the image height, w is the image width, m and n are the variables in the summation, the value range of m is (1, h), the value range of n is (1, w), and U c It is the feature map of the cth channel in the multi-scale feature U.
[0097] Then, two one-dimensional convolutions are introduced to fuse all statistical information and integrate the interdependence between channels in the feature vector S, thereby enhancing the information expression of a certain scale feature map. In this process, there is a certain dimensional scaling, and the feature vector output by the second one-dimensional convolution is reshaped into a score matrix of size (3, c). The score matrix is mapped to the weight coefficient matrix of the three convolution kernels through softmax calculation. The sum of the three values in each column of the weight coefficient matrix is 1, which corresponds to the weights of the three sizes of features U1, U2, and U3 output by the three convolution kernels at the channel. The weight coefficient matrix is obtained by learning by the SCKB module, and the most appropriate weights are automatically assigned to the three sizes of features U1, U2, and U3 corresponding to the three different convolution kernels, without manual setting. Finally, the weighted values of each feature map and the corresponding weights are calculated to obtain multi-scale fusion features. Then, after a BN layer, the maximum pooling layer is used to filter out high-dimensional features with robustness.
[0098] The specific process of step 4 includes the following steps:
[0099] The shallow features extracted by the encoding path (including the multi-scale features extracted by the last two layers) are input into the decoding path to extract deep features, while optimizing the expression of the change information of the feature map. The decoding path consists of four UM modules. That is, the input of the first UM module is the superposition of the encoding results of the two encoding paths, and the input features of the following three UM modules are the direct superposition of the output features of the previous UM module and the output features of the two encoding paths of the same level. The upsampling module UM with an attention mechanism is used to extract the deep features in the U-Net network, and selectively fuse them with the shallow features extracted by the encoding path;
[0100] Specifically, for the encoding paths on both sides, the features obtained by superimposing the fourth-layer convolutional network are superimposed with the output of the third-layer convolutional network after passing through the upsampling module UM, and then pass through the upsampling module UM; then superimposed with the output of the second-layer convolutional network and then pass through the upsampling module UM; then the output of the upsampling module UM is superimposed with the output of the first-layer convolutional network and then passes through the upsampling module UM to obtain the final change feature map, and finally a 1×1 two-dimensional convolution is performed to obtain the final change detection result map.
[0101] like Figure 4 As shown in the figure, since the input features of the upsampling module UM are obtained by simply superimposing shallow features and deep features, it is necessary to use a channel attention module (CAM) to optimize the input features in the channel dimension. The importance of each channel can be encoded in a one-dimensional channel weight vector. Each channel weight coefficient is automatically learned by the network. The specific calculation process is as follows:
[0102] M c =σ(MLP(AvgPool(F2))+MLP(MaxPool(F2)))
[0103] In the formula, F2 represents the input feature, M c Represents the weight vector of channel attention, MLP represents multi-layer perceptron, and σ represents the Sigmoid activation function.
[0104] First, perform maximum pooling and average pooling on F2 in the spatial dimension to obtain two feature vectors with the same length as the number of F2 channels, and send these two feature vectors to two multilayer perceptrons (MLPs) respectively. Then, add and fuse the two output results at the pixel level, that is, the feature vector after maximum pooling passes through two MLPs to obtain one output result, and the feature vector after average pooling also passes through these two MLPs to obtain another output result, and then add and fuse these two results;
[0105] Then, based on the Sigmoid activation function, the addition result of the multilayer perceptron is mapped to a weight vector between zero and one, and the value on each weight vector represents the importance of the corresponding feature channel.
[0106] Finally, the weight vector and input features are reorganized, and the optimization method of channel attention reorganization features is as follows:
[0107]
[0108] in, represents the optimized output features, F2 represents the input features, and M c The weight vector representing the channel attention, Represents tensor product, which is the multiplication of elements at corresponding positions of two tensors.
[0109] After the input features of the upsampling module UM are optimized in the channel dimension by CAM, they are immediately sent to two consecutive 3×3 convolutional layers to extract deep features.
[0110] Before the deep feature is passed to the next upsampling module, in order to enable the subsequent transposed convolution layer to learn more significant change features from the feature map, the spatial attention mechanism is also needed to optimize and reorganize the deep feature map in the pixel dimension. Spatial attention encodes the information at each pixel position in the feature map, and then the neural network layer adaptively learns the spatial attention map. The calculation method of the spatial attention map is as follows:
[0111] M s=σ(Conv2D([AvgPool(F4);MaxPool(F4)]))
[0112] In the formula, F4 represents the input feature, M s Represents the spatial attention map. After obtaining the spatial attention map, the spatial reorganization feature is calculated as follows:
[0113]
[0114] Where F4 represents the input feature, M s represents the spatial attention map, Represents tensor product, which is the multiplication of elements at corresponding positions of two tensors.
[0115] Channel attention enables the selective fusion of shallow and deep features in the U-Net results, while spatial attention suppresses the feature information of non-changing pixels and enhances the difference features of changing pixels.
[0116] After four UM modules, the output features of the last UM module pass through a convolution layer with a 1×1 convolution kernel to adjust the number of channels of the final change detection map. The output result of the 1×1 convolution layer is the change detection result of the network.
[0117] The specific process of step 5 includes the following steps:
[0118] The multi-spectral change detection method based on multi-scale adaptive convolution kernel can already realize the classification and positioning of changed pixels, but there is still the problem of inaccurate positioning due to information loss. To address this problem, the present invention uses a multimodal conditional random field to perform a secondary detection on the change detection result of the output of the neural network in step 4.
[0119] The overall process, such as Figure 5 As shown in the figure, sample equalization and sample enhancement are first used to reduce the impact of sample imbalance on MSAK-Net, and then MSAK-Net is trained in an end-to-end manner and the network weights are saved. After that, the test image is sent to the trained network to obtain the network's prediction result and the unary potential function of the FC-CRF model is constructed with the result; at the same time, the CVA difference map and the SA spectral difference map are calculated for the test image, and the two difference maps are used to construct the binary potential function of the FC-CRF model, so as to achieve the purpose of fine classification and obtain the final change detection result image.
[0120] Fully Connected Conditional Random Field (FC-CRF) is an optimization of the conditional random field. It overcomes the limitation of the conditional random field that does not have long-range dependencies by establishing the connection relationship between all pixels in the image, and can better restore local information.
[0121] The probability distribution function of a conditional random field conforms to the Gibbs distribution, and the Gibbs distribution is calculated by the product of a series of non-negative energy functions of maximal groups in the undirected graph model. Therefore, the probability distribution of the FC-CRF output result Y can be defined as follows:
[0122]
[0123] Where X represents the input, p1 and q1 are intermediate variables for summation, ranging from 1 to N, and φ u represents the univariate potential function, φ p represents the binary potential function. φ u It is usually calculated by the Softmax classification result output by the neural network, and the formula is:
[0124]
[0125] It indicates the probability strength that the neural network considers pixel p2 to be a changed pixel. Based on the performance of the neural network, which contains more noise points and discontinuities, it is necessary to introduce a binary potential function to consider the positional relationship between pixels. This method uses multimodal information as the input information of FC-CRF and proposes a new binary potential function to calculate the secondary classification result. The redefined binary potential function is expressed as:
[0126]
[0127] in, and are the binary potential functions defined for the grayscale difference map extracted by change vector analysis (CVA) and the spectral difference map calculated by spectral angle (SA), respectively. and are the weights of the two potential functions, p2 and q2 are the intermediate variables for summation. For example, the detailed calculation formula is:
[0128]
[0129] is the label function, l is the summed intermediate variable, K represents the number of Gaussian kernels, and only when and The binary potential function will only take effect when they are two different pixel points. The mathematical expression is:
[0130]
[0131] w (l) is the Gaussian kernel The weight coefficient of and It is the feature vector corresponding to pixel points p2 and q2.
[0132] In the change detection task, a double Gaussian kernel is generally used. The Gaussian kernel is defined as follows:
[0133]
[0134] in Respectively represent the position vectors of pixels p2 and q2, They respectively represent the difference intensity values of pixels p2 and q2 in the CVA difference map.
[0135] The first Gaussian kernel in the above formula is used to define whether adjacent pixels with similar grayscale values in the difference map belong to the same category, σ α With σ β is the Gaussian kernel parameter, which indicates the degree of proximity and similarity of pixels; the second Gaussian kernel is used to smooth the boundaries and noise of the classification results, and the smoothing effect is determined by the parameter σ γ Determine, w1 and w2 are the weights of the two Gaussian kernels.
[0136] The calculation process and The same, except that it uses the SA spectral difference map to define the grayscale vector.
[0137] Finally, the mean field approximation algorithm is used to iteratively optimize the energy potential function of the FC-CRF model to derive the category label corresponding to each pixel.
[0138] Two ways to calculate the difference map:
[0139] When scholars previously constructed the binary potential function, they used the grayscale vector of the original image to construct it, that is, I x =(i x1 ,i x2 ,…,i xN ) and I y =(i y1 ,i y2 ,…,i yN ) These two vectors, N is the number of channels, so in the calculation When , since it is a vector, we need to find its Euclidean distance In the present invention, two difference maps are used to construct the binary potential function, and the two difference maps are two grayscale images, each pixel has only one value, that is, Therefore, in the present invention, the CVA difference map measures the intensity of change by the size difference of the grayscale vector between the two-phase images. Assume that I x =(i x1 ,i x2 ,…,i xN ) and I y =(iy1 ,i y2 ,…,i yN ) represents the grayscale vector of the two images at a certain location, then the calculation formula of the difference intensity of the CVA difference map at that location is:
[0140]
[0141] In the formula, i1 is the intermediate variable for summation.
[0142] For point p2, CVA(I x ,I y )The formula at pixel p2 is as follows
[0143]
[0144] When using, It is the difference intensity value in the CVA difference map, that is
[0145] Spectral angle (SA) treats the spectrum of each pixel position in the two original images as a high-dimensional vector, and measures the similarity of the pixel at that position in the spectral dimension by the angle between the two spectral vectors. Assume V x =(v x1 , v x2 , …, v xn ) and V y =(v y1 , v y2 , …, v yn ) is the spectral vector of the pixels in the same position of the two images, then SA(V x ,V y ) is calculated as:
[0146]
[0147] Where SA(V x ,V y ) ranges from 0 to π. The smaller the value of the spectral angle, the higher the similarity between the two feature vectors, and the greater the possibility that they belong to the same type of features. i2 is the intermediate variable for summation. The spectral angle measures the magnitude of the change in terms of the similarity of the feature vectors.
[0148] The CVA difference map contains the grayscale difference information between the pixels of the dual-phase multispectral image, while the SA spectral difference map extracts the spectral difference information from the angle difference of the spectral vector. Different from using only a single geometric difference map to construct the conditional random field, the two difference maps extract the local change features in the original image from different angles, and input them into the FC-CRF model at the same time to complement each other, so that the secondary detection result map has more accurate change boundary positioning and less noise.
[0149] The effect diagram obtained by using the present invention is as follows Figure 6-Figure 9 As shown, Figure 6-Figure 9 The detection results of the present invention on two multispectral data sets are: Figure 6-Figure 9 The first column is the image before the change, the second column is the image after the change, the third column is the true value map of the change, and the fourth column is the detection result of the proposed method.
[0150] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multispectral image change detection method based on multi-scale adaptive convolution kernel, characterized in that: The multispectral image is input into the network MASK-Net with weight-shared bilateral encoding paths for detection; The network MASK-Net of the bilateral encoding path simultaneously extracts independent features of the dual-phase multispectral image; each encoding path includes four layers of convolutional networks connected in sequence, and a maximum pooling layer is set between every two layers of the four-layer convolutional network; the first two layers of the convolutional network in the four-layer convolutional network each include two convolutional layers with 3×3 convolutional kernels and one BN layer; the last two layers of the convolutional network each include two SCKB modules and one BN layer; Then, the shallow features extracted by the encoding path are input into the decoding path to extract deep features. The decoding path includes four upsampling modules. The input of the first upsampling module is formed by superimposing the encoding results of the two encoding paths. The input features of the following three upsampling modules are directly superimposed by the output features of the previous upsampling module and the output features of the two encoding paths of the same level. The upsampling module with an attention mechanism is used to extract the deep features in the network, and selectively fuses them with the shallow features extracted by the encoding path. After passing through four upsampling modules, the output features of the last upsampling module are passed through a convolution layer with a 1×1 convolution kernel to adjust the number of channels of the final change detection map, and output the change detection result. The SCKB module is a convolution kernel selection module based on adaptive weights; the SCKB module is divided into three convolution branches, each of which includes a convolution layer, a BN layer, and an activation layer. The convolution kernel sizes in the convolution layers of the three convolution branches are 3×3, 5×5, and 7×7, respectively, which are used to extract features U1, U2, and U3 of three sizes; the features U1, U2, and U3 based on the three sizes are superimposed on the channel dimension to obtain a multi-scale feature map U: U=[U1;U2;U3]=[Conv 3×3 (F1);Conv 5×5 (F1);Conv 7×7 (F1)] Among them, Conv() represents the convolution operation, F1 is the input of the SCKB module; [;] in [U1; U2; U3] means that these three feature maps are superimposed on the channel dimension through the concat operation; Then, global information is encoded through global pooling to generate a feature vector S in the channel dimension. The first third of the feature vector S represents the statistical information corresponding to the 3×3 convolution kernel, the second third represents the statistical information corresponding to the 7×7 convolution kernel, and the middle represents the statistical information corresponding to the 5×5 convolution kernel. Then, two one-dimensional convolutions are introduced to fuse all statistical information, and the feature vector output by the second one-dimensional convolution is reshaped into a score matrix; the score matrix is mapped to the weight coefficient matrix of the three convolution kernels through softmax calculation, and each column in the weight coefficient matrix corresponds to the weight of the three-size features U1, U2, and U3 output by the three convolution kernels at the channel; finally, the weighted values of U1, U2, and U3 and the corresponding weights are calculated to obtain multi-scale fusion features; then, after a BN layer, the maximum pooling layer is used to obtain high-dimensional features.
2. The method for detecting changes in multispectral images based on a multiscale adaptive convolution kernel according to claim 1, characterized in that: The cth element of the feature vector S is calculated as follows: In the formula, h is the image height, w is the image width, m and n are the variables in the summation, the value range of m is (1, h), the value range of n is (1, w), U c It is the feature map of the cth channel in the multi-scale feature U.
3. The method for detecting changes in multispectral images based on a multiscale adaptive convolution kernel according to claim 2, characterized in that: The convolutional layers in the first two convolutional networks of the encoding path are convolutional layers with activation functions, and the activation function is Relu.
4. The method for detecting changes in multispectral images based on a multiscale adaptive convolution kernel according to claim 2, characterized in that: The upsampling module is the UM module. The UM module first performs maximum pooling and average pooling on the input feature F2 in the spatial dimension to obtain two feature vectors with the same length as the number of F2 channels, and respectively sends the two feature vectors to two multi-layer perceptrons MLP, and then adds and fuses the two output results at the pixel level; then, based on the Sigmoid activation function, the addition result of the multi-layer perceptron is mapped to a weight vector between zero and one, that is, the weight vector M of the channel attention. c ; The weight vector and input features are reorganized, and the optimization method of the channel attention reorganization feature is as follows: in, represents the optimized output features, Represents tensor product, that is, the multiplication of elements at corresponding positions of two tensors; Then it is sent to two consecutive 3×3 convolutional layers to obtain feature F4; The spatial attention map is then adaptively learned by the neural network layer: M s =σ(Conv2D([AvgPool(F4);MaxPool(F4)])) Where M s represents the spatial attention map; Conv2D represents the two-dimensional convolution; σ represents the Sigmoid activation function; After obtaining the spatial attention map, the spatial reorganization feature is then calculated; the calculation method of the spatial reorganization feature is as follows: Where M s represents the spatial attention map, Represents tensor product, which is the multiplication of elements at corresponding positions of two tensors.
5. The method for detecting changes in multispectral images based on a multiscale adaptive convolution kernel according to claim 4, characterized in that: The MASK-Net is a trained MASK-Net, which uses weighted cross entropy as the loss function during training: Where N is the total number of samples, t i Indicates the label of the i-th pixel. When the pixel belongs to the change area, t i is 1, otherwise, the pixel is a non-changing pixel. i is 0; i Represents the prediction result of the neural network for the i-th pixel, y i The larger the value, the more likely the neural network thinks that the pixel belongs to the change area. i represents the probability of the neural network predicting a non-changing pixel; w c Represents the category weight.
6. The method for detecting changes in multispectral images based on a multiscale adaptive convolution kernel according to claim 5, characterized in that: The multispectral image input to the network MASK-Net of the weight-shared bilateral encoding path is obtained by segmenting the entire multispectral image.
7. The method for detecting changes in multispectral images based on a multi-scale adaptive convolution kernel according to claim 6, characterized in that: The training set used to train MASK-Net needs to be processed with data enhancement. During the enhancement, the samples are selectively enhanced. If the proportion of changed pixels in the training sample is less than 1%, it is filtered so that it does not participate in the network training. If the proportion of changed pixels in the training sample is more than 5%, image enhancement is used for it. Each training sample is rotated and flipped 90°, 180°, and 270°. The training sample is then sent to the network for training.
8. A multispectral image change detection method based on a multi-scale adaptive convolution kernel according to any one of claims 1 to 7, characterized in that: The method also includes the step of performing secondary detection using a multimodal conditional random field, and the specific process includes the following steps: The multispectral image is input into MASK-Net for detection to obtain the prediction result of the network, and the unary potential function of the FC-CRF model is constructed with the prediction result; at the same time, the CVA difference map and the SA spectrum difference map are calculated for the multispectral image, and the two difference maps are used to construct the binary potential function of the FC-CRF model; and then a secondary detection is performed through the fully connected conditional random field; Fully connected conditional random field is FC-CRF. The probability distribution of the FC-CRF output result Y is defined by the following formula: Where X represents the input, p1 and q1 are intermediate variables for summation, ranging from 1 to N, and φ u represents the unary potential function, φ p represents the binary potential function; φ u The Softmax classification result output by the neural network is calculated using the formula: It indicates the probability strength that the neural network believes that pixel p2 belongs to the changed pixel; The binary potential function is expressed as: in, and They are Based on the binary potential function defined by the grayscale difference map extracted by the change vector analysis CVA and the spectral difference map calculated by the spectral angle SA, and are the weights of the two potential functions, p2 and q2 are the intermediate variables for summation; The formula is as follows: in, is the label function, l is the summed intermediate variable, and K represents the number of Gaussian kernels; The Gaussian kernel is as follows: in Respectively represent the position vectors of pixels p2 and q2, Respectively represent the difference intensity values of pixels p2 and q2 in the CVA difference map; σ α With σ β is the Gaussian kernel parameter in the previous Gaussian kernel; σ γ is the Gaussian kernel parameter in the previous Gaussian kernel; w1 and w2 are the Gaussian kernel weights; For the SA spectrum difference map, the processing process is the same as same; Finally, the mean field approximation algorithm is used to iteratively optimize the energy potential function of the FC-CRF model to derive the category label corresponding to each pixel.
9. The method for detecting changes in multispectral images based on a multi-scale adaptive convolution kernel according to claim 8, characterized in that: When constructing the binary potential function, the calculation formula for the difference intensity of the CVA difference map at this point is: In the formula, i1 is the intermediate variable for summation; For point p2, CVA(I x , I y )The formula at pixel p2 is as follows: In the formula, It is the difference intensity value in the CVA difference map, that is Let V x =(v x1 , v x2 , ..., v xn ) and V y =(v y1 , v y2 , ..., v yn ) is the spectral vector of the pixels in the same position of the two images, then SA(V x , V y ) is as follows: Among them, i2 is the summation intermediate variable; SA(V x , V y ) ranges from 0 to π. The smaller the value of the spectral angle, the higher the similarity between the two feature vectors, and the greater the possibility that they belong to the same type of features. The calculation process and Similarly, in order to distinguish the physical meaning of the parameters in the calculation, In Denoted as It is used to represent the angular difference of the spectral vectors of pixels p2 and q2 in the SA difference map; For point p2, It is the angle difference of the spectral vector in the SA difference map, 10. The method for detecting changes in multispectral images based on a multi-scale adaptive convolution kernel according to claim 9, characterized in that: The label function As follows: Among them, w (l) is the Gaussian kernel The weight coefficient of and It is the feature vector corresponding to pixel points p2 and q2.
