A building change detection method based on multi-scale twin networks

Abstract

The present invention discloses a method for detecting building changes based on a multi-scale twin network, comprising: performing color space conversion on two images containing building changes to be detected; performing brightness histogram matching; performing chromaticity histogram matching; constructing a multi-scale twin network model and training it; using the trained model to perform multi-scale feature extraction on the image to obtain a feature map; combining the feature map with the trained model to obtain a difference map, thereby completing building change detection based on a multi-scale twin network. In order to solve the problem of false detection caused by shadows on buildings and other landmark coverings, the present invention adopts a double histogram matching method to eliminate the differences that affect chromaticity and brightness, and at the same time introduces a multi-scale feature extraction model to reduce the scale differences between the changed buildings. This enables preliminary screening of renovation and expansion sites for important targets such as buildings, and assists in detailed and precise judgment of changed areas.

Description

Technical Field

[0001] The present invention relates to a building change detection method, in particular to a building change detection method based on a multi-scale twin network. Background Art

[0002] Existing change detection methods primarily include pixel-level and object-level change detection methods. Pixel-level change detection methods use pixels as the analysis unit and employ direct comparison or post-classification comparison strategies. These methods are effective for low- and medium-resolution remote sensing imagery. While simple and easy to implement, they face difficulties in determining the optimal threshold and are unable to identify the type of change. Object-level change detection methods can be categorized as those based on image set segmentation and those based on independent image segmentation. These methods are simple and easy to implement, but are unable to detect objects with changing shapes. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a building change detection method based on a multi-scale twin network in response to the shortcomings of the existing technology.

[0004] In order to solve the above technical problems, the present invention discloses a building change detection method based on a multi-scale twin network, comprising the following steps:

[0005] Step 1: performing color space conversion on two images containing building changes to be detected, namely, the image at the first moment and the image at the second moment;

[0006] Step 2: perform brightness histogram matching, i.e., use the brightness channel of the image at the first moment to normalize the brightness channel of the image at the second moment;

[0007] Step 3, perform chromaticity histogram matching, that is, use the chromaticity channel of the image at the second moment to standardize the chromaticity channel of the image at the first moment;

[0008] Step 4: Build a multi-scale twin network model;

[0009] Step 5: training the multi-scale twin network model;

[0010] Step 6: Using the trained multi-scale twin network model, perform multi-scale feature extraction on the image at the first moment to form a first feature map; perform multi-scale feature extraction on the image at the second moment to form a second feature map;

[0011] In step 7, the first feature map and the second feature map are used to obtain a difference map using the trained multi-scale twin network model to complete building change detection based on the multi-scale twin network.

[0012] The method for color space conversion in step 1 of the present invention is:

[0013] Convert from RGB space to YCbCr space, including:

[0014] Y=0.299R+.587G+0.114B

[0015] Cb=0.564(BY)

[0016] Cr=0.713(RY)

[0017] Among them, Y represents the brightness value of the image, Cb represents the blue density offset of the image, Cr represents the red density offset of the image, R represents the red channel value of the image, G represents the green channel value of the image, and B represents the blue of the image.

[0018] The method for performing brightness histogram matching in step 2 of the present invention includes:

[0019] Step 2-1, convert the first moment image and the second moment image into a first brightness image l1(x, y) and a second brightness image l2(x, y):

[0020] l1(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0021] l2(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0022] Among them, R (x，y) 、B (x，y) and G (x，y) is the RGB value of the pixel point (x, y), where x represents the x-axis coordinate of the pixel point and y represents the y-axis coordinate of the pixel point;

[0023] Step 2-2: Based on the brightness at the first moment, match the brightness at the second moment to the brightness at the first moment:

[0024]

[0025] The method for performing chromaticity histogram matching in step 3 of the present invention includes:

[0026] Step 3-1, convert the first moment image and the second moment image into a first chromaticity image S1(x, y) and a second chromaticity image S2(x, y):

[0027] S1(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0028] S2(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0029] Step 3-2: Using the chromaticity of the image at the second moment as a reference, match the chromaticity of the image at the first moment to the brightness of the image at the second moment:

[0030]

[0031] The multi-scale twin network model described in step 4 of the present invention includes a multi-scale feature extraction network and a measurement network.

[0032] The multi-scale feature extraction network includes: four convolutional layers and one pooling layer;

[0033] Among them, the convolution kernel size of the first convolution layer is 1*1, and the number of convolution kernels is 24; the second convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 6; the third convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 12; the fourth convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 18; the fifth layer is the pooling layer, the pooling window size is 2*2, and the stride is 2*2;

[0034] The measurement network consists of three fully connected layers. The number of neurons in the first two fully connected layers is 128, and the activation function adopts the relu function (reference: R Hahnloser, R. Sarpeshkar, MA Mahowald, RJ Douglas, HS Seung (2000). "Digital selection and analogue amplification coexist in acortex-inspired silicon circuit". Nature 405:947-951.); the number of neurons in the third fully connected layer is 2, and the activation function adopts the softmax function (reference: Li Li, Ying Sancong. Implementation of Softmax Layer of Convolutional Neural Network Based on FPGA [J]. Modern Computer (Professional Edition), 2017 (26): 21-24.).

[0035] The method for training the multi-scale twin network model in step 5 of the present invention includes:

[0036] During the training process, the cross entropy loss function L is used. log (y,p):

[0037] L log (y,p)=-(ylog(p)+(1-y)log(1-p))

[0038] Where y is the label (1 for the changed class and 0 for the non-changed class), and p is the predicted probability;

[0039] During the training process, the gradient descent method is used to optimize the network parameters. The gradient descent is defined as follows:

[0040]

[0041] Among them, J(θ) is the given loss function, m is the number of samples input for each training, and h θ (x i ) is the weight of the training sample, x i is the training sample value, y i is the label value of the sample, and i is the sequence number of the sample.

[0042] The method of performing multi-scale feature extraction on the first moment image in step 6 of the present invention to form a first feature map includes:

[0043] Perform a 1*1 convolution operation on the image at the first moment to form the first part of the first feature map;

[0044] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 6 on the image at the first moment to form the second part of the first feature map;

[0045] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 12 on the image at the first moment to form the third part of the first feature map;

[0046] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 18 on the image at the first moment to form the fourth part of the first feature map;

[0047] Performing a pooling operation on the image at the first moment to form the fifth part of the first feature map;

[0048] The five parts of the first feature map are connected and then a 1*1 convolution operation is performed to form the first feature map, that is, the first feature map vector.

[0049] The method of performing multi-scale feature extraction on the image at the second moment to form a second feature map in step 6 of the present invention includes:

[0050] Perform a 1*1 convolution operation on the image at the second moment to form the first part of the second feature map;

[0051] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 6 on the image at the second moment to form the second part of the second feature map;

[0052] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 12 on the image at the second moment to form the third part of the second feature map;

[0053] Perform a dilated convolution with a kernel of 3*3 and a dilated number of 18 on the image at the second moment to form the fourth part of the second feature map;

[0054] Performing a pooling operation on the image at the second moment to form the fifth part of the second feature map;

[0055] The five parts of the second feature map are connected and then a 1*1 convolution operation is performed to form the second feature map, that is, the second feature map vector.

[0056] The method for obtaining the difference map described in step 7 of the present invention includes:

[0057] The distance between the first feature map and the second feature map is measured; using the trained multi-scale twin network model and a measure learning method, a distance metric reflecting the similarity difference information between phase images at different times is obtained, the changed area is separated from the non-changed area, and the distance between the pixels in the non-changed area is minimized to obtain a difference map of the building changes.

[0058] Beneficial effects:

[0059] The building change detection method based on multi-scale twin networks proposed in this project uses a method that combines two-time histogram matching and multi-scale twin networks to effectively solve the problems caused by large differences in chromaticity and brightness between two-phase data, shadows on buildings, and large size differences. Compared with existing pixel-level and target-level algorithms, the detection accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0061] Figure 1 Schematic diagram of the method of the present invention.

[0062] Figure 2 Schematic diagram of the multi-scale twin network structure in the present invention.

[0063] Figure 3 This is an image of a certain area in November in the embodiment.

[0064] Figure 4 This is an image of a certain area in April in the embodiment.

[0065] Figure 5 for Figure 3 and Figure 4The result of applying this method to change detection. DETAILED DESCRIPTION

[0066] This paper proposes a building change detection method based on a multi-scale twin network. To address the problem of false detection caused by large differences in chromaticity and brightness between two-temporal images, large differences in the scale of the buildings being changed, building shadows, and other surface cover, histogram matching is used in the preprocessing stage to correct chromaticity and brightness. The corrected remote sensing images at the two moments are fed into the multi-scale twin network for training, generating a difference map.

[0067] like Figure 1 As shown in FIG, a building change detection method based on a multi-scale twin network includes the following steps:

[0068] Step 1: Convert the image at time 1 and the image at time 2 from RGB space to YCbCr space.

[0069] Y=0.299R+.587G+0.114B

[0070] Cb=0.564(BY)

[0071] Cr=0.713(RY)

[0072] Step 2: Perform brightness histogram matching, and use the brightness channel at time 1 to regulate the brightness channel at time 2. Step 2 includes:

[0073] Step 2-1: Convert the time 1 image and the time 2 image into luminance images l1(x, y) and l2(x, y).

[0074] l1(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0075] l2(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0076] where R (x，y) 、B (x，y) , G (x，y) is the RGB value of the corresponding pixel (x, y).

[0077] Step 2-2: Based on the brightness of moment 1, match the brightness of moment 2 to that of moment 1.

[0078]

[0079] Step 3: Perform chromaticity histogram matching and use the chromaticity channel at time 2 to normalize the chromaticity channel at time 1.

[0080] It is characterized in that the step 3 comprises:

[0081] Step 3-1: Convert the image at time 1 and the image at time 2 into chrominance images S1(x, y) and S2(x, y)

[0082] S1(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0083] S2(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0084] Step 3-2: Based on the brightness at time 2, match the brightness at time 1 to that at time 2.

[0085]

[0086] Step 4: Perform multi-scale feature extraction on the image at time 1 to form a feature map. It is characterized in that step 4 includes:

[0087] Step 4-1: Perform a 1*1 convolution operation on the image at time 1 to form features Figure 1-1 .

[0088] Step 4-2: Perform convolution with kernel 3*3 and hole number 6 on the image at time 1 to form the feature Figure 1-2 .

[0089] Step 4-3: Perform convolution with kernel 3*3 and hole number 12 on the image at time 1 to form the feature Figure 1-3 .

[0090] Step 4-4: Perform convolution with kernel 3*3 and hole number 18 on the image at time 1 to form the feature Figure 1-4 .

[0091] Step 4-5: Perform pooling operation on the moment 1 image to form features Figure 1-5 .

[0092] Step 4-6: Add features Figure 1-1 , 1-2, 1-3, 1-4, and 1-5 are connected, and then a 1*1 convolution operation is performed to form a feature map vector.

[0093] Step 5: performing multi-scale feature extraction on the moment 2 image to form a feature map, wherein step 5 includes:

[0094] Step 5-1: Perform a 1*1 convolution operation on the moment 2 image to form features Figure 2-1 .

[0095] Step 5-2: Perform convolution with kernel 3*3 and hole number 6 on the image at time 2 to form the feature Figure 2-2 .

[0096] Step 5-3: Perform convolution with kernel 3*3 and hole number 12 on the image at time 2 to form the feature Figure 2-3 .

[0097] Step 5-4: Perform convolution with kernel 3*3 and hole number 18 on the image at time 2 to form the feature Figure 2-4 .

[0098] Step 5-5: Perform pooling operation on the moment 2 image to form features Figure 2-5 .

[0099] Step 5-6: Add features Figure 2-1 , 2-2, 2-3, 2-4, and 2-5 are connected, and then a 1*1 convolution operation is performed to form a feature map vector.

[0100] Step 6: Measure the distance between the feature map of the image at time 2 and the feature map of the image at time 2. By using metric learning, we can mine the similarity rules contained in the data and learn a distance metric that can reflect the similarity and difference information between multi-temporal images. This effectively separates the changed areas from the non-changed areas and minimizes the distance between the pixels in the non-changed areas as much as possible to form a change difference map. The metric network consists of three fully connected layers.

[0101] The number of neurons in the first two fully connected layers is 128, and the activation function is the relu function; the number of neurons in the third fully connected layer is 2, and the activation function is the softmax function.

[0102] Step 7: After the model is built, the model must be trained first. During the training process, the cross entropy loss function is used. It is defined as:

[0103] L log (y,p)=-(ylog(p)+(1-y)log(1-p))

[0104] Where y is the label (1 for the changed class and 0 for the non-changed class), and p is the predicted probability

[0105] During the training process, the gradient descent method is used to optimize the network parameters. The gradient descent is defined as follows: Given a loss function

[0106]

[0107] Where m is the number of samples input for each training, h θ (x i ) is the weight of the training sample.

[0108] Step 8: After the model training is completed, the model parameters are used to perform step-by-step calculations to obtain the difference map.

[0109] Example 1:

[0110] This paper provides a building change detection method based on a multi-scale twin network. Taking building change detection classification as an example, we first construct a change detection dataset. For example, we select the SECOND semantic change detection dataset, where each image is 512*512 in size. 100 pairs of images are selected as the training set, and 20 pairs of images are selected as the test set for testing. The specific change detection steps are as follows:

[0111] Step 1: Convert each image in the training set to a color space from RGB space to YCbCr space.

[0112] Y=0.299R+.587G+0.114B

[0113] Cb=0.564(BY)

[0114] Cr=0.713(RY)

[0115] Step 2: Select one image from each pair of images in the training set as image 1 and the other as image 2, and use the brightness channel of image 1 to normalize the brightness channel of image 2.

[0116] l1(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0117] l2(x, y)=0.31×R (x，y) +0.29×B (x，y) +0.4×G (x，y)

[0118]

[0119] Step 3: Normalize the chroma channel of image 1 using the chroma channel of image 2:

[0120] S1(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0121] S2(x, y)=0.21×R (x，y) +0.72×B (x，y) +0.07×G (x，y)

[0122]

[0123] Step 4: Construct a multi-scale feature extraction network, which mainly consists of four convolutional layers and one pooling layer. The convolution kernel size of the first convolutional layer is l*1, the number of convolution kernels is 24, the second convolutional layer uses dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 6. The third convolutional layer uses dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 12. The fourth convolutional layer uses dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 18. The fifth pooling layer has a pooling window size of 2*2 and a stride of 2*2.

[0124] Step 5: After step 4, concatenate the output vectors of image 1 and image 2 in each image pair to form a feature map.

[0125] Step 6: Construct a metric network consisting of three fully connected layers. The number of neurons in the first two fully connected layers is 128, the number of neurons in the third fully connected layer is 2, and the activation function uses the softmax function.

[0126] Step 7: After the model is built, the cross entropy loss function and gradient descent method are used to train the model.

[0127] Step 8: After the model training is completed, the model parameters are used to perform step-by-step calculations to obtain the difference map.

[0128] Example 2:

[0129] like Figure 3 As shown, a real map of a certain area to be detected is cut out from Google Maps, and the map is a map of November 2021, which serves as the first moment image described in the method of the present invention;

[0130] like Figure 4 As shown, the image is taken from Google Maps and Figure 3 The same real map of the area to be inspected in a certain place is shown, and the map is a map of April 2022, which serves as the second moment image in the method of the present invention;

[0131] After testing by the method of the present invention, the test results are obtained, such as Figure 5 As shown, the black part represents the area where no change is detected, and the white part represents the area where changes are detected, which mainly reflect the changes in buildings and farmland in the embodiment.

[0132] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium is capable of storing a computer program that, when executed by the data processing unit, can execute the invention of a building change detection method based on a multi-scale twin network provided by the present invention and some or all of the steps in each embodiment. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0133] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a computer program, i.e., a software product. The computer program software product can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0134] This invention provides a concept and method for building change detection based on a multi-scale twin network. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the invention, and such improvements and modifications should also be considered within the scope of protection of the invention. Any components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A building change detection method based on a multi-scale twin network, characterized in that: The steps include: Step 1: performing color space conversion on two images containing building changes to be detected, namely, the image at the first moment and the image at the second moment; Step 2: perform brightness histogram matching, i.e., use the brightness channel of the image at the first moment to normalize the brightness channel of the image at the second moment; Step 3, perform chromaticity histogram matching, that is, use the chromaticity channel of the image at the second moment to standardize the chromaticity channel of the image at the first moment; Step 4: Build a multi-scale twin network model; Step 5: training the multi-scale twin network model; Step 6: Using the trained multi-scale twin network model, perform multi-scale feature extraction on the image at the first moment to form a first feature map; perform multi-scale feature extraction on the image at the second moment to form a second feature map; Step 7: Using the trained multi-scale twin network model, the first feature map and the second feature map are used to obtain a difference map, thereby completing building change detection based on the multi-scale twin network. The multi-scale twin network model in step 4 includes a multi-scale feature extraction network and a measurement network; The multi-scale feature extraction network in step 4 includes: four convolutional layers and one pooling layer; Among them, the convolution kernel size of the first convolution layer is 1*1, and the number of convolution kernels is 24; the second convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 6; the third convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 12; the fourth convolution layer adopts dilated convolution, the convolution kernel size is 3*3, the number of convolution kernels is 64, and the number of holes is 18; the fifth layer is the pooling layer, the pooling window size is 2*2, and the stride is 2*2; The measurement network described in step 4 consists of three fully connected layers. The number of neurons in the first two fully connected layers is 128, and the activation function uses the relu function; the number of neurons in the third fully connected layer is 2, and the activation function uses the softmax function; The method for obtaining the difference map described in step 7 includes: The distance between the first feature map and the second feature map is measured; using the trained multi-scale twin network model and a measure learning method, a distance metric reflecting the similarity difference information between phase images at different times is obtained, the changed area is separated from the non-changed area, and the distance between the pixels in the non-changed area is minimized to obtain a difference map of the building changes.

2. A building change detection method based on a multi-scale twin network according to claim 1, characterized in that: The method for color space conversion in step 1 is: Convert from RGB space to YCbCr space, including: Y=0.299R+.587G+0.114B Cb=0.564(BY) Cr=0.713 (RY) Among them, Y represents the brightness value of the image, Cb represents the blue density offset of the image, Cr represents the red density offset of the image, R represents the red channel value of the image, G represents the green channel value of the image, and B represents the blue of the image.

3. The building change detection method based on multi-scale twin network according to claim 2 is characterized in that: The method for performing brightness histogram matching in step 2 includes: Step 2-1: Convert the first moment image and the second moment image into a first brightness image and the second brightness image : ； in, Pixel RGB value, x represents the x-axis coordinate of the pixel, and y represents the y-axis coordinate of the pixel; Step 2-2: Based on the brightness at the first moment, match the brightness at the second moment to the brightness at the first moment: 。 4. The building change detection method based on a multi-scale twin network according to claim 3 is characterized in that: The method for performing chromaticity histogram matching in step 3 includes: Step 3-1: Convert the first moment image and the second moment image into a first chromaticity image and the second chrominance image : ； Step 3-2: Using the chromaticity of the image at the second moment as a reference, match the chromaticity of the image at the first moment to the brightness of the image at the second moment: 。 5. The building change detection method based on multi-scale twin network according to claim 4 is characterized in that: The method for training the multi-scale twin network model in step 5 includes: During the training process, the cross entropy loss function is used : ； in, For labels, is the predicted probability; During the training process, the gradient descent method is used to optimize the network parameters. The gradient descent is defined as follows: ； in, For a given loss function, m is the number of samples input for each training, is the weight of the training sample, is the training sample value, is the label value of the sample, and i is the sequence number of the sample.

6. The building change detection method based on multi-scale twin network according to claim 5 is characterized in that: The method of performing multi-scale feature extraction on the first moment image in step 6 to form a first feature map includes: Perform a 1*1 convolution operation on the image at the first moment to form the first part of the first feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 6 on the image at the first moment to form the second part of the first feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 12 on the image at the first moment to form the third part of the first feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 18 on the image at the first moment to form the fourth part of the first feature map; Performing a pooling operation on the image at the first moment to form the fifth part of the first feature map; The five parts of the first feature map are connected and then a 1*1 convolution operation is performed to form the first feature map, that is, the first feature map vector.

7. The building change detection method based on multi-scale twin network according to claim 6 is characterized in that: The method of performing multi-scale feature extraction on the image at the second moment to form a second feature map in step 6 includes: Perform a 1*1 convolution operation on the image at the second moment to form the first part of the second feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 6 on the image at the second moment to form the second part of the second feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 12 on the image at the second moment to form the third part of the second feature map; Perform a dilated convolution with a kernel of 3*3 and a dilated number of 18 on the image at the second moment to form the fourth part of the second feature map; Performing a pooling operation on the image at the second moment to form the fifth part of the second feature map; The five parts of the second feature map are connected and then a 1*1 convolution operation is performed to form the second feature map, that is, the second feature map vector.

Images