Image prediction method, device and medium of building change detection model

Abstract

The application relates to the technical field of change detection, and specifically discloses an image prediction method and device of a building change detection model and a medium, which comprises the following steps: acquiring a first image used for representing a building before change and a second image used for representing a building after change; inputting the first image and the second image into an encoder for feature extraction to obtain a first feature image and a second feature image; converging the first feature image and the second feature image into a stream to obtain a feature stream, inputting the feature stream into a multi-scale feature fusion module for feature fusion to obtain a feature fusion image; inputting the feature fusion image into a first decoder for feature decoding and difference identification to obtain a feature fusion decoding image; and inputting the feature fusion decoding image into a second decoder to obtain a prediction image. The multi-scale feature fusion module fuses and detects multi-layer features, ensures that small-scale change buildings are not missed, and improves the accuracy of the prediction image of the change buildings.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of change detection, and particularly relates to an image prediction method of a building change detection model, equipment and a medium. BACKGROUND

[0002] Change detection (CD) is a process of identifying the "significant differences" between multi-temporal remote sensing images, which can easily distinguish different types of changes and quantitatively analyze the degree of change. Building change detection is one of the important directions of remote sensing application, which provides very important information in land use change detection, urban landscape design, environmental monitoring and rapid response to disaster events. Rapid and accurate building change detection is also of great significance in urban construction, planning and other urban management.

[0003] The building change detection method can be roughly divided into two categories: one is based on traditional image processing, and the other is based on deep learning. The method based on traditional image processing is divided into three levels: pixel level, feature level and target level image processing. This method involves more human participation, and the results have many pseudo-change areas. For example, the pixel-level detection method only considers the features of the original single pixel, and does not contain the spatial relationship information of adjacent pixels, which is easy to be disturbed by noise; and some information may be lost in the feature extraction process.

[0004] However, the current deep neural network has some problems in the detection performance of the changed building. For example, when FCSC, FCSD and Unet++, MSOF network are used for building detection, the boundary of the building is not processed finely enough, and the detection result is sticky. In addition, FCSC is a U-shaped double-branch structure, which concatenates the feature maps of each layer in each branch for jump connection. CDNet is a single-branch network, which merges channels when the image is input into the network, and uses jump connection for feature enhancement, so that the accuracy of the detection result is not enough; the integrity detection ability of large buildings needs to be improved, and there are false detection rate and missed detection rate phenomena. SUMMARY

[0005] The following is a summary of the subject matter described in detail in this document. This summary is not intended to limit the scope of protection of the claims.

[0006] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a building change detection model image prediction method, device and medium, which are advantageous in solving the problems of insufficient fine processing of the boundary of the building and adhesion of the detection result in related detection technologies, and reducing the false detection rate and the missed detection rate phenomenon of large building detection. The building change detection model image prediction method in the present application fuses the multi-layer features of the image through a multi-scale feature fusion module and then detects, improves the completeness of the effective information of the image of the changed building, ensures that the small-scale changed building is not missed, improves the accuracy of the predicted image of the changed building, and also ensures the generalization performance and robustness of the network model. In addition, the network is trained by setting a deep supervision branch in the building change detection model, which alleviates the gradient vanishing of the building change detection model and further improves the accuracy of the building change detection model in predicting the changed building image.

[0007] In a first aspect, the embodiments of the present application provide a building change detection model image prediction method, which comprises: obtaining a first image used to represent a building before a change and a second image used to represent a building after a change; inputting the first image and the second image into an encoder for feature extraction to obtain a first feature image and a second feature image; converging the first feature image and the second feature image into a stream to obtain a feature stream, inputting the feature stream into a multi-scale feature fusion module for feature fusion to obtain a feature fusion image; inputting the feature fusion image into a first decoder for feature decoding and difference identification to obtain a feature fusion decoding image; and inputting the feature fusion decoding image into a second decoder to obtain a predicted image.

[0008] The technical solution of the first aspect of the present application has at least one of the following advantages or beneficial effects: by obtaining a first image used to represent a building before a change and a second image used to represent a building after a change, the original features of each individual double-phase image can be preserved, the first image and the second image are input into an encoder for feature extraction, and the obtained feature stream is converged into a stream and input into a multi-scale feature fusion module for multi-layer feature fusion and detection, which improves the completeness of the effective information of the image of the changed building, ensures that the small-scale changed building is not missed, and improves the accuracy of the predicted image of the changed building; the feature fusion image is subjected to feature decoding and difference identification through the first decoder and the second decoder, which further improves the completeness of the effective information of the image and the accuracy of the predicted image of the changed building.

[0009] Further, the building change detection model is a double-branch end-to-end twin network, the double-branch end-to-end twin network comprises an encoder, a multi-scale feature fusion module, a decoder and a deep supervision branch, an output end of the encoder is connected with an input end of the multi-scale feature fusion module, an output end of the multi-scale feature fusion module is connected with an input end of the decoder, an output end of the decoder is connected with an input end of the deep supervision branch, and the deep supervision branch is used for relieving gradient disappearance of the building change detection model.

[0010] Further, the first decoder is an attention decoding block, the attention decoding block comprises a channel attention mechanism and a spatial attention mechanism; the inputting the feature fusion image into the first decoder for feature decoding and difference identification to obtain a feature fusion decoding image comprises:

[0011] inputting the feature fusion image into the channel attention mechanism to obtain a channel weight feature, and combining the feature fusion image with the channel weight feature and a residual connection to obtain a first enhanced feature image;

[0012] performing feature fusion on the first enhanced feature image through a 3x3 convolution layer to obtain a first feature fusion image,

[0013] performing channel compression on the first feature fusion image through a 1x1 convolution layer and conveying the first feature fusion image to the spatial attention mechanism to obtain a spatial attention feature, and combining the spatial attention feature with a residual connection to obtain a second enhanced feature image;

[0014] conveying the second enhanced feature image to two 3x3 convolution layers to extract a change feature to obtain a change feature image;

[0015] conveying the change feature image to the deep supervision branch and a transposed convolution, and performing up-sampling on the change feature image through the transposed convolution to obtain a feature fusion decoding image.

[0016] Further, feature images Stage1, Stage2, Stage3 and Stage4 output by different layers in the encoder are connected to the attention decoding block with the same size through a jump connection to supplement image features of the single first image or the second image.

[0017] Further, the attention decoding block comprises a first attention decoding block, a second attention decoding block, a third attention decoding block and a fourth attention decoding block; and the inputting the feature fusion image into the first decoder for feature decoding and difference identification to obtain a feature fusion decoding image further comprises:

[0018] fuse the feature fusion image and Stage4 of the first image, Stage4 of the second image into a feature flow, and input the feature flow into the first attention decoding block to obtain a first global change image;

[0019] fuse the first global change image and Stage3 of the first image, Stage3 of the second image into a feature flow, and input the feature flow into the second attention decoding block to obtain a second global change image;

[0020] fuse the second global change image and Stage2 of the first image, Stage2 of the second image into a feature flow, and input the feature flow into the third attention decoding block to obtain a third global change image;

[0021] fuse the third global change image and Stage1 of the first image, Stage1 of the second image into a feature flow, and input the feature flow into the fourth attention decoding block to obtain a feature fusion decoding image.

[0022] Further, the step length and padding of global pooling in the encoder are 1 and 2 respectively, and the input of the first image and the second image into the encoder for feature extraction to obtain a first feature image and a second feature image comprises:

[0023] input the first image and the second image into an encoder with shared structure and parameters for feature extraction to obtain a first feature image and a second feature image, the first feature image and the second feature image have the same size; the sizes of the feature images Stage1, Stage2, Stage3, Stage4 and Stage5 output by different layers in the encoder are 256x256, 128x128, 64x64, 32x32 and 16x16 respectively.

[0024] Further, the multi-scale feature fusion module comprises five parallel branches, the first branch comprises a 1x1 convolution layer, the second, third and fourth branches each comprise a 3x3 atrous convolution layer with rates of 2, 4 and 6 respectively, and the fifth branch comprises a global pooling layer and an up-sampling layer; the input of the feature flow into the multi-scale feature fusion module for feature fusion to obtain a feature fusion image comprises:

[0025] input the feature flow into the five parallel branches for feature extraction to obtain a first branch feature image, a second branch feature image, a third branch feature image, a fourth branch feature image and a fifth branch feature image;

[0026] The first branch feature image, the second branch feature image, the third branch feature image, the fourth branch feature image and the fifth branch feature image are channel cascaded to obtain a fusion feature;

[0027] The fusion feature is input into a 1x1 convolution layer to adjust the channel number, and then a 1x1 convolution layer and a 3x3 empty convolution layer are used to cooperate with a residual connection to extract features, so as to obtain a preliminary feature fusion image;

[0028] The preliminary feature fusion image is input into the full pooling layer and the up-sampling layer to be compressed to 1x1, so as to obtain a global feature;

[0029] The global feature is input into a 1x1 convolution layer to extract features and reduce dimensions, so as to obtain a feature fusion image.

[0030] Further, the feature fusion decoding image is input into a second decoder to obtain a prediction image, including:

[0031] The feature fusion decoding image is input into the second decoding block, and the feature fusion decoding image is extracted by the second decoder and compressed in channels to obtain a feature fusion decoding image;

[0032] A probability map of each pixel point change of the feature fusion decoding image is obtained by using a Sigmoid function, and the probability map is threshold segmented to obtain a prediction image.

[0033] In a second aspect, an electronic device is also provided, including: a memory for storing computer program code; and a processor for executing the computer program code stored in the memory to implement the image prediction method of the building change detection model according to the first aspect.

[0034] In a third aspect, a readable storage medium is also provided, which stores a computer program, and the computer program is executed by a processor to implement the method of the building change detection model according to the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a step flowchart of the image prediction method of the building change detection model provided by the embodiments of the present application;

[0036] Figure 2 is a structural schematic diagram of a double-branch end-to-end twin network in the building change detection model provided by the embodiments of the present application;

[0037] Figure 3 is Figure 1 is a step flowchart of step S500 in the building change detection model.

[0038] Figure 4 is Figure 1 Another step flow chart of step S500 in the embodiment;

[0039] Figure 5 is Figure 1 Step flow chart of step S200 in the embodiment;

[0040] Figure 6 is Figure 1 Step flow chart of step S400 in the embodiment;

[0041] Figure 7 is Figure 1 Step flow chart of step S600 in the embodiment;

[0042] Figure 8 is Figure 2 Connection diagram of deep supervision branch in the embodiment;

[0043] Figure 9 is a hardware structure schematic diagram of an electronic device provided by the embodiment. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0045] In the description of the present application, plural means more than two. If there is a description of first, second, it is only used to distinguish technical features for the purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features.

[0046] Referring to Figure 1 , Figure 1 is a step flow chart of an image prediction method of a building change detection model provided by the embodiment; comprising obtaining a first image for representing a building change before and a second image for representing a building change after; inputting the first image and the second image into an encoder for feature extraction, to obtain a first feature image and a second feature image; converging the first feature image and the second feature image into a stream to obtain a feature stream, inputting the feature stream into a multi-scale feature fusion module for feature fusion to obtain a feature fusion image; inputting the feature fusion image into a first decoder for feature decoding and difference identification to obtain a feature fusion decoding image; inputting the feature fusion decoding image into a second decoder to obtain a predicted image.

[0047] By acquiring a first image for characterizing a building before change and a second image for characterizing a building after change, the original features of each single dual-phase image can be preserved, the first image and the second image are input to an encoder for feature extraction, the obtained feature stream is input to a multi-scale feature fusion module for fusion of multi-layer features and then detection, the completeness of effective information of the image of the changed building is improved, the small-scale changed building is not missed, and the accuracy of the predicted image of the changed building is improved; the feature fusion image is subjected to feature decoding and difference identification through the first decoder and the second decoder, the completeness of the effective information of the image is further improved, and the accuracy of the predicted image of the changed building is improved.

[0048] With reference to Figure 2 , Figure 2 is a structural diagram of a dual-branch end-to-end twin network in a building change detection model provided by an embodiment of the present application. In the embodiment of the present application, the building change detection model is a dual-branch end-to-end twin network 700, which comprises an encoder, a multi-scale feature fusion module, a decoder and a deep supervision branch. The output end of the encoder is connected to the input end of the multi-scale feature fusion module, the output end of the multi-scale feature fusion module is connected to the input end of the decoder, the output end of the decoder is connected to the input end of the deep supervision branch, and the deep supervision branch is used to relieve the gradient vanishing of the building change detection model.

[0049] It should be noted that the dual-branch end-to-end twin network 700 in the building change detection model is based on the depth and width of the network. In the embodiment of the present application, the encoder is selected to be ResNet34, and the type of the encoder is not limited in the embodiment of the present application.

[0050] It should be noted that in the embodiment of the present application, the multi-scale feature fusion module in the dual-branch end-to-end twin network 700 selects MSFF for multi-scale feature fusion. Instead of using only the feature image of the last layer for detection, the MSFF selects multiple layers of features for fusion and then detection to ensure the completeness of the effective information of the feature image. The type of the multi-scale feature fusion module is not limited in the embodiment of the present application.

[0051] With reference to Figure 3 , Figure 3 is Figure 1 is a step flowchart of step S500 in FIG. 5. Step S500 comprises inputting the feature fusion image to the first decoder for feature decoding and difference identification to obtain a feature fusion decoding image. The first decoder is an attention decoding block, which comprises a channel attention mechanism and a spatial attention mechanism. Step S500 specifically comprises:

[0052] Step S510: input the feature fusion image into the channel attention mechanism to obtain the weight feature of the channel, and combine the feature fusion image with the weight feature of the channel and the residual connection to obtain a first enhanced feature image;

[0053] Step S520: performing feature fusion on the first enhanced feature image through a 3x3 convolution layer to obtain a first feature fusion image;

[0054] Step S530: performing channel compression on the first feature fusion image through a 1x1 convolution layer and sending it to the spatial attention mechanism to obtain a spatial attention feature, and combining the spatial attention feature with the residual connection to obtain a second enhanced feature image;

[0055] Step S540: sending the second enhanced feature image to two 3x3 convolution layers to extract change features and obtain a change feature image;

[0056] Step S550: sending the change feature image to the depth supervision branch and the transposed convolution, and performing up-sampling on the change feature image through the transposed convolution to obtain a feature fusion decoding image.

[0057] By inputting the feature fusion image into the channel attention mechanism, the weight feature of the channel distribution can be obtained. By introducing the attention mechanism, the connection between any two positions on the feature map is established. For the features at a position on the feature map, the feature information at all positions on the feature map can be calculated through weighted summation, and then the first enhanced feature image is obtained through residual connection, which ensures the integrity of the effective information of the small-scale change building. The first enhanced feature image is fused through a 3x3 convolution layer to obtain a first feature fusion image, which further identifies the effective information of the first enhanced feature image. The first enhanced feature image is sent to a 1x1 convolution layer for channel compression and sent to the spatial attention mechanism to obtain a spatial attention feature. The spatial attention feature is combined with the residual connection to obtain a second enhanced feature image. The main purpose of introducing the spatial attention mechanism is to assign different weights to different spatial positions on the feature map, so as to emphasize the different importance of different spatial positions in feature extraction. After adding the spatial attention mechanism, the spatial position weight of important features can be increased, so that the important features are more effective, the integrity of the effective information of the change building image is improved, and the precision of the change building detection is improved. At the same time, it will not increase the calculation amount too much, and will not affect the calculation speed of the network model.

[0058] Referring to Figure 4 , Figure 4 is Figure 1 another step flowchart of step S500 in the method. Step S500 includes inputting the feature fusion image into the first decoder for feature decoding and difference identification to obtain a feature fusion decoding image. Step S500 specifically includes:

[0059] Step S560: splice and input the feature fusion image and the feature flow fused from Stage4 of the first image and Stage4 of the second image to the first attention decoding block to obtain a first global change image;

[0060] Step S570: splice and input the first global change image and the feature flow fused from Stage3 of the first image and Stage3 of the second image to the second attention decoding block to obtain a second global change image;

[0061] Step S580: splice and input the second global change image and the feature flow fused from Stage2 of the first image and Stage2 of the second image to the third attention decoding block to obtain a third global change image;

[0062] Step S590: splice and input the third global change image and the feature flow fused from Stage1 of the first image and Stage1 of the second image to the fourth attention decoding block to obtain a feature fusion decoding image.

[0063] The early layers in the encoder, i.e. Stage1, Stage2, Stage3 and Stage4 of the first image and the second image, contain lower-level local structure information of the images, which are connected to the attention decoding blocks with the same scale through the skip connection of Stage1, Stage2, Stage3 and Stage4 of the first image and the second image to supplement the features of the single first image or the second image, to generate the first, second, third and fourth global change images with compact size, and the second, third and fourth attention decoding blocks all take the output of the previous attention decoding block and the splicing result of the corresponding Stage as the input of the next attention decoding block, which improves the completeness of the effective information of the changed building image, ensures that the small-scale changed building is not missed, and improves the accuracy of the predicted image of the changed building.

[0064] Referring to Figure 5 , Figure 5 is Figure 1 the step flow chart of step S200 in FIG. 2; step S200 includes inputting the feature flow to the multi-scale feature fusion module for feature fusion to obtain a feature fusion image. Wherein, the step length and padding of the global pooling in the encoder are 1 and 2 respectively. Step S200 specifically includes:

[0065] Step S210: input the first image and the second image to the encoder with shared structure and parameters for feature extraction to obtain a first feature image and a second feature image;

[0066] Step S220: the sizes of Stage1, Stage2, Stage3, Stage4, and Stage5 in the first feature image and the second feature image output by different layers in the encoder are 256x256, 128x128, 64x64, 32x32, and 16x16 respectively.

[0067] The first image and the second image are input into the encoder with shared structure and parameters for feature extraction to obtain the first feature image and the second feature image, wherein the sizes of Stage1, Stage2, Stage3, Stage4, and Stage5 in the first feature image and the second feature image are the same. The encoder with shared structure and parameters used in the embodiment of the present application reduces the amount of parameters and the amount of calculation, is not prone to overfitting, is beneficial to saving computing power and making the building change detection model smaller. By sharing the same weights and biases, the process of learning new parameters of the hidden layer neurons of each branch is reduced, and the learning efficiency is improved.

[0068] Referring to Figure 6 , Figure 6 is Figure 1 a step flowchart of step S400 in FIG. 4; step S400 includes inputting the feature stream into a multi-scale feature fusion module for feature fusion to obtain a feature fusion image. The multi-scale feature fusion module includes five parallel branches, the first branch includes a 1x1 convolutional layer, the second, third, and fourth branches each include a 3x3 dilated convolutional layer with a rate of 2, 4, and 6 respectively, and the fifth branch includes a global pooling layer and an upsampling layer. Step S400 specifically includes:

[0069] Step S410: inputting the feature stream into the five parallel branches for feature extraction to obtain a first branch feature image, a second branch feature image, a third branch feature image, a fourth branch feature image, and a fifth branch feature image;

[0070] Step S420: channel concatenation of the first branch feature image, the second branch feature image, the third branch feature image, the fourth branch feature image, and the fifth branch feature image to obtain a fusion feature;

[0071] Step S430: inputting the fusion feature into a 1x1 convolutional layer to adjust the number of channels, and then using a 1x1 convolutional layer and a 3x3 dilated convolutional layer in combination with a residual connection for feature extraction to obtain a preliminary feature fusion image;

[0072] Step S440: inputting the preliminary feature fusion image into a global pooling layer and an upsampling layer to compress to 1x1 to obtain a global feature;

[0073] Step S450: input the global feature into a 1x1 convolution layer for feature extraction and dimension reduction to obtain a feature fusion image.

[0074] The multi-scale feature fusion module mainly consists of an Atrous spatial pyramid pooling (ASPP) and includes five parallel branches. The first branch includes a 1x1 convolution layer, the second, third and fourth branches respectively include a 3x3 atrous convolution layer with rates of 2, 4 and 6, and the fifth branch includes a global pooling layer and an up-sampling layer. The feature stream is input into the five parallel branches for multi-scale feature fusion, wherein the kernel size of the atrous convolution layer is always kept at 3x3, and the convolution layer can expand the receptive field of the convolution kernel without losing resolution, which can extract semantic information of different receptive fields in semantic segmentation, realize free multi-scale feature extraction and improve segmentation accuracy. The preliminary feature fusion image is compressed by the two-dimensional adaptive global average pooling layer and the up-sampling layer to obtain a global feature, and then the global feature is input into a 1x1 convolution layer for further feature extraction and dimension reduction to obtain a feature fusion image.

[0075] The calculation process of the ASPP is as follows:

[0076]

[0077] wherein f 1×1 represents a 1x1 convolution operation, and respectively represent atrous convolution operations with atrous rates of 2, 4 and 6, up represents an up-sampling operation, and glo pool represents a global average pooling.

[0078] The multi-scale feature fusion module in the embodiment of the application no longer uses only the feature map of the last layer for detection, but selects multiple layers of features for fusion and then detection, thereby ensuring the completeness of the effective information of the feature map, improving the completeness of the effective information of the changed building image, ensuring that small-scale changed buildings are not missed, and improving the accuracy of the prediction image of the changed building.

[0079] Referring to Figure 7 , Figure 7 is Figure 1 a step flowchart of step S600 in the method; step S600 includes inputting the feature fusion decoding image into a second decoder to obtain a prediction image; and step S600 specifically includes:

[0080] Step S610: input the feature fusion decoding image into a second decoding block to extract features from the feature fusion decoding image and compress channels by using the second decoder, thereby obtaining a feature fusion decoding image;

[0081] Step S620: Obtain a probability map of each pixel point change of the feature fusion decoding image by using a Sigmoid function, and perform threshold segmentation on the probability map to obtain a prediction image.

[0082] The second decoder does not have an attention mechanism, and the second decoder includes a 1x1 convolution layer and a Sigmoid function, the feature fusion decoding image is input into the 1x1 convolution layer in the second decoder to extract and compress channels, and the feature fusion decoding image is obtained; the Sigmoid function is used to obtain a probability map of each pixel point change of the feature fusion decoding image, and threshold segmentation is performed on the probability map to obtain a prediction image, thereby improving the accuracy of the changed building prediction image.

[0083] With reference to Figure 8 , Figure 8 is Figure 2 a connection diagram of the depth supervision branch. In the embodiment of the present application, the building change detection model is a double-branch end-to-end twin network 700, which introduces three depth supervision branches with the same structure, namely DS1, DS2 and DS3. The output end of the first attention decoding block is connected to the input end of DS1, the output end of the second attention decoding block is connected to the input end of DS2, and the output end of the third attention decoding block is connected to DS3. It should be noted that the change map generated by the depth supervision branch is only used for effective training of the double-branch end-to-end twin network 700, and the final prediction image result is obtained in the decoding stage of the second decoder.

[0084] Depth supervision branch DS i The calculation formula of the depth supervision branch is as follows:

[0085] DS i =σ(f 1x1 (ADM i ))

[0086] Wherein, ADM i represents the image difference feature calculated by the attention decoding block. Sigma represents the sigmoid function, and f 1×1 represents a 1x1 convolution operation. In the training process, the loss of each depth supervision is calculated independently and directly back-propagated to the intermediate layer. In this way, the intermediate layer in the network is effectively trained, and the weights of the intermediate layer can be finely updated, thereby alleviating the problem of gradient disappearance.

[0087] With reference to Figure 9 , Figure 9is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application, comprising a processor, which can be realized in the mode of a general CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used for executing a related program to realize the privacy data processing method based on vertical federated learning provided by the embodiment of the present application; a memory, which can be realized in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory) etc. The memory can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are realized by software or firmware, the related program codes are saved in the memory and are called and executed by the processor to realize the embodiment of the present application; an input / output interface, which is used for realizing information input and output; a communication interface, which is used for realizing the communication interaction between the device and other devices, and can realize the communication in the wired mode (for example, a USB, a network cable, etc.) or the wireless mode (for example, a mobile network, WIFI, Bluetooth, etc.); and a bus, which is used for transmitting information between various components (for example, the processor, the memory, the input / output interface, and the communication interface) of the device. The processor, the memory, the input / output interface, and the communication interface realize the communication connection between each other in the device through the bus.

[0088] The embodiment of the present application further provides a storage medium, the storage medium is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to realize the flowchart of the privacy data processing method based on vertical federated learning. The memory is a non-transitory computer readable storage medium, which can be used for storing a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0089] The image prediction method of the building change detection model, the electronic device and the medium provided by the embodiment of the application, by acquiring a first image used for representing a building change before and a second image used for representing a building change after, inputting the first image and the second image into an encoder for feature extraction to obtain a first feature image and a second feature image, converging the first feature image and the second feature image into a stream to obtain a feature stream, inputting the feature stream into a multi-scale feature fusion module for feature fusion to obtain a feature fusion image, inputting the feature fusion image into a first decoder for feature decoding and difference identification to obtain a feature fusion decoding image, and inputting the feature fusion decoding image into a second decoder to obtain a prediction image. In the embodiment, the building change detection model is a double-branch end-to-end twin network 700, the double-branch end-to-end twin network 700 includes an encoder, a multi-scale feature fusion module, a decoder and a deep supervision branch, the output end of the encoder is connected with the input end of the multi-scale feature fusion module, the output end of the multi-scale feature fusion module is connected with the input end of the decoder, the output end of the decoder is connected with the input end of the deep supervision branch, and the deep supervision branch is used for relieving gradient disappearance of the building change detection model.

[0090] By acquiring the first image used for representing the building change before and the second image used for representing the building change after, the original features of each individual double-phase image can be retained, the first image and the second image are input into the encoder for feature extraction to obtain a feature stream, the feature stream is converged into a stream, the multi-layer features are fused and detected again by inputting the feature stream into the multi-scale feature fusion module, the completeness of the effective information of the image of the changed building is improved, the small-scale changed building is not missed, and the accuracy of the prediction image of the changed building is improved. The feature fusion image is input into the first decoder and the second decoder for feature decoding and difference identification, the completeness of the effective information of the image is further improved, the accuracy of the prediction image of the changed building is improved, the deep supervision branch is arranged in the double-branch end-to-end twin network 700, the gradient disappearance of the building change detection model is effectively relieved, and the accuracy of the prediction image of the changed building is further improved.

[0091] The preferred embodiments of the embodiments of the application are described above with reference to the accompanying drawings, and the scope of the right of the embodiments of the application is not limited by this. Any modification, equivalent replacement and improvement made by the person skilled in the art without departing from the scope and essence of the embodiments of the application should be within the scope of the right of the embodiments of the application.

Claims

1. An image prediction method for a building change detection model, characterized in that, include: Acquire a first image to characterize the building before the change and a second image to characterize the building after the change; The first image and the second image are respectively input into the encoder for feature extraction to obtain the first feature image and the second feature image; The first feature image and the second feature image are merged into a stream to obtain a feature stream; The feature stream is input to a multi-scale feature fusion module for feature fusion to obtain a feature fused image; The feature-fused image is input into the first decoder for feature decoding and difference recognition to obtain the feature-fused decoded image; The feature-fused decoded image is input into the second decoder to obtain the predicted image; The multi-scale feature fusion module includes five parallel branches. The first branch includes a 1x1 convolutional layer; the second, third, and fourth branches each contain a 3x3 dilated convolutional layer with speeds of 2, 4, and 6, respectively; and the fifth branch includes a global pooling layer and an upsampling layer. The process of inputting the feature stream into the multi-scale feature fusion module for feature fusion to obtain a fused feature image includes: The feature stream is input to the five parallel branches for feature extraction, resulting in the first branch feature image, the second branch feature image, the third branch feature image, the fourth branch feature image, and the fifth branch feature image. The first branch feature image, the second branch feature image, the third branch feature image, the fourth branch feature image, and the fifth branch feature image are concatenated to obtain fused features; The fused features are input into a 1x1 convolutional layer to adjust the number of channels, and then a 1x1 convolutional layer and a 3x3 dilated convolutional layer are used in conjunction with residual connections to extract features, thus obtaining a preliminary feature fusion image. The preliminary feature fusion image is input into the global pooling layer and the upsampling layer and compressed to 1x1 to obtain global features; The global features are input into a 1x1 convolutional layer for feature extraction and dimensionality reduction to obtain a feature fusion image.

2. The image prediction method for the building change detection model according to claim 1, characterized in that, The building change detection model is a two-branch end-to-end Siamese network. The two-branch end-to-end Siamese network includes an encoder, a multi-scale feature fusion module, a decoder, and a deep supervision branch. The output of the encoder is connected to the input of the multi-scale feature fusion module, the output of the multi-scale feature fusion module is connected to the input of the decoder, and the output of the decoder is connected to the input of the deep supervision branch. The deep supervision branch is used to mitigate the gradient vanishing problem in the building change detection model.

3. The image prediction method for the building change detection model according to claim 2, characterized in that, The first decoder is an attention decoding block, which includes a channel attention mechanism and a spatial attention mechanism; the step of inputting the feature fusion image into the first decoder for feature decoding and difference recognition to obtain a feature fusion decoded image includes: The feature fusion image is input into the channel attention mechanism to obtain the channel weight features. The feature fusion image is then combined with the channel weight features and residual connections to obtain the first enhanced feature image. The first enhanced feature image is passed through a 3x3 convolutional layer for feature fusion to obtain the first feature fused image; The first feature fusion image is passed through a 1x1 convolutional layer for channel compression and then fed into the spatial attention mechanism to obtain spatial attention features. The spatial attention features are combined with residual connections to obtain the second enhanced feature image. The second enhanced feature image is fed into two 3x3 convolutional layers to extract the change features, thus obtaining the change feature image; The changed feature image is fed into the deep supervision branch and the transposed convolution, and the transposed convolution upsamples the changed feature image to obtain a feature fusion decoded image.

4. The image prediction method for the building change detection model according to claim 2, characterized in that, The feature images Stage1, Stage2, Stage3, and Stage4 output by different layers in the encoder are connected to attention decoding blocks of the same size via skip connections to supplement the image features of a single first image or second image.

5. The image prediction method for the building change detection model according to claim 3, characterized in that, The attention decoding block includes a first attention decoding block, a second attention decoding block, a third attention decoding block, and a fourth attention decoding block; the step of inputting the feature fusion image into the first decoder for feature decoding and difference recognition to obtain the feature fusion decoded image further includes: The feature stream formed by fusing the feature fusion image with Stage 4 of the first image and Stage 4 of the second image is concatenated and input into the first attention decoding block to obtain the first global change image; The feature stream formed by fusing the first global change image, Stage 3 of the first image, and Stage 3 of the second image is concatenated and input into the second attention decoding block to obtain the second global change image; The feature stream formed by fusing the second global change image, Stage 2 of the first image, and Stage 2 of the second image is concatenated and input into the third attention decoding block to obtain the third global change image; The feature stream formed by fusing the third global change image, Stage 1 of the first image, and Stage 1 of the second image is concatenated and input into the fourth attention decoding block to obtain the feature fusion decoded image.

6. The image prediction method for the building change detection model according to claim 1, characterized in that, The encoder uses a global pooling step of 1 and padding of 2. The step of inputting the first image and the second image into the encoder for feature extraction to obtain a first feature image and a second feature image includes: The first image and the second image are respectively input into an encoder with a shared structure and parameters for feature extraction to obtain a first feature image and a second feature image. The first feature image and the second feature image have the same size. The sizes of the feature images Stage1, Stage2, Stage3, Stage4 and Stage5 output by the encoder at different layers are 256×256, 128×128, 64×64, 32×32 and 16×16, respectively.

7. The image prediction method for the building change detection model according to claim 1, characterized in that, The step of inputting the feature-fused decoded image into a second decoder to obtain a predicted image includes: The feature fusion decoded image is input into the second decoding block, and the second decoder performs feature extraction and channel compression on the feature fusion decoded image to obtain the feature fusion decoded image; The Sigmoid function is used to obtain the probability map of the changes of each pixel in the feature fusion decoded image, and the probability map is thresholded to obtain the predicted image.

8. An electronic device, characterized in that, include: Memory, used to store computer program code; A processor for executing the computer program code stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images