Building roof tree shielding repairing method and device, electronic equipment and storage medium

Abstract

The invention provides a building roof tree shielding restoration method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a tree shielding image and a Gaussian pure noise image of a to-be-restored building roof; repeatedly executing the following steps until a preset step length is reached: predicting a sheltering position image and a roof prediction image of the tree sheltering image in the current time step based on a tree sheltering decomposition model; based on the sheltering position image, the roof prediction image and the tree sheltering image to be repaired, the repair change constraint from the current time step to the next time step is determined; performing conditional control denoising on the Gaussian pure noise image based on the restoration change constraint to obtain a conditional diffusion restoration image, and performing fusion restoration on the conditional diffusion restoration image based on a tree occlusion fusion restoration sampler to obtain a tree occlusion image in the next time step; the tree shielding image is a complete building roof image in the last time step. According to the invention, the accuracy of building roof extraction in the tree shielding environment can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic equipment and storage medium for repairing tree occlusion on a building roof. Background Art

[0002] As one of the important symbols of social development, buildings are closely related to human life. With the rapid development of remote sensing earth observation technology, large-scale high-resolution remote sensing images provide accurate and reliable basic data for building change monitoring, urban planning and construction, disaster emergency assessment and other applications. At present, when using remote sensing images for large-scale building information extraction tasks, problems such as errors and missed detection of buildings blocked by trees are still a challenging task.

[0003] This method focuses on tree occlusion, which is quite common but rarely studied as a focus. Trees that block roofs block the inherent extension of building boundaries. These trees and their shadows change the spectrum of roof pixels, which poses a great challenge to accurately depict buildings. The current common research method is to optimize the deep network structure to improve the extraction accuracy of building roofs under tree occlusion, without giving priority to restoration. Since the image segmentation model predicts the building roof at the spatial pixel level. This is difficult to solve the problem of inaccurate roof morphology extraction caused by tree occlusion. In actual engineering application projects, in order to obtain CityGML LOD0 result data that meets GIS quality standards, a lot of subsequent quality inspection work such as manual visual interpretation is still required. Some studies use multi-source data to make up for the limitations of a single optical data source. For example, additional radar point clouds that can penetrate the canopy can be used to extract missing roof information. However, the acquisition cost of large-scale and spatially aligned multi-source data is too high. Graph neural networks improve the reasoning ability of boundaries under occlusion by judging the number and spatial relationship of roof corners, and can directly generate vector results. However, when the roof is occluded in a large area, the information of the existing edges or corners is incomplete, and the usability of the vector contour boundary obtained is not high. The above methods all start from the information of the visible part of the roof, and have limited ability to infer the information of the occluded part of the roof.

[0004] Generative models have been widely used in the field of image restoration. For example, the generative adversarial network optimizes the adversarial loss between the generator and the discriminator to make the generated image as close to the real image as possible. After the encoder part obtains the image features of the visible part, the decoder is used to repair the occluded part of the image. Therefore, when facing a large area of ​​tree occlusion, the area of ​​the roof occluded by the trees can be repaired first, and then extracted.

[0005] However, there are two key problems in repairing tree shading on roofs that are difficult to solve: 1. How to determine the location of trees in the image that interfere with the extraction of building roofs. The core of this problem is that not all trees in the image need to be identified as obstructions. How to determine the location of tree obstructions; how to perform targeted repairs according to different degrees of obstruction.

[0006] 2. How to use the information of the visible area of ​​the building roof to repair the occluded part. The core of this problem is whether the model can understand the pixel distribution of the complete building roof. The roof is repaired by using the image's own features and its spatial layout in the background. Summary of the invention

[0007] The present invention provides a method, device, electronic device and storage medium for repairing tree occlusion on a building roof, so as to solve the problem that the current deep learning method extracts inaccurate information on the building roof when the building roof is blocked by trees in high spatial resolution remote sensing images.

[0008] The present invention provides a method for repairing tree shading on a building roof, comprising: Obtain a tree occlusion image and a Gaussian pure noise image of the roof of the building to be repaired, wherein the tree occlusion image at the first time step is the tree occlusion image of the roof of the building to be repaired; Repeat the following steps until a preset step length is reached: based on the tree occlusion decomposition model, predict the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain the tree occlusion image of the next time step; The tree occlusion image at the last time step is used as the final restored complete building roof image.

[0009] According to the method for repairing tree occlusion on the roof of a building provided by the present invention, the tree occlusion decomposition model includes an encoder, a position decoder and a pixel decoder. The tree occlusion decomposition model is used to predict the occlusion position image of the tree occlusion image at the current time step, and the roof prediction image of the tree occlusion image at the current time step includes: Based on the encoder, extracting intermediate transition features of the tree occlusion image at the current time step; Based on the position decoder and the intermediate transition features, predicting an occlusion position image of the tree occlusion image at the current time step; Based on the pixel decoder and the intermediate transition features, a roof prediction image of the tree occlusion image at the current time step is predicted.

[0010] According to the building roof tree occlusion repair method provided by the present invention, the occlusion position image of the tree occlusion image at the current time step is predicted based on the position decoder and the intermediate transition feature, including: Based on the position decoder, calculating an attention score of the intermediate transition feature; Multi-head self-attention calculation is performed on intermediate transition features whose attention scores are greater than or equal to a preset threshold, and convolution calculation is performed on intermediate transition features whose attention scores are less than the preset threshold. The calculation results are fused to obtain the occlusion position image of the tree occlusion image at the current time step.

[0011] According to the method for repairing tree occlusion on the roof of a building provided by the present invention, the prediction of the roof prediction image of the tree occlusion image at the current time step based on the pixel decoder and the intermediate transition feature includes: Based on the pixel decoder, the intermediate transition features are subjected to partial random masking to obtain masked features and unmasked features; A convolution calculation is performed on the masked features, a multi-head self-attention calculation is performed on the unmasked features, and the calculation results are fused to obtain a roof prediction image of the tree-occluded image at the current time step.

[0012] According to the building roof tree occlusion repair method provided by the present invention, the loss function of the tree occlusion decomposition model is determined based on the positioning loss function, the prediction loss function and the repair loss function; The positioning loss function is determined based on the occlusion position image of the tree occlusion image at the current time step and the sample true occlusion position image, the prediction loss function is determined based on the roof prediction image of the tree occlusion image at the current time step and the sample building roof image, and the restoration loss function is determined based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step and the sample building roof image.

[0013] According to the method for repairing tree occlusion on a building roof provided by the present invention, the tree occlusion repair change constraint from the current time step to the next time step is determined based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the building roof to be repaired, including: Determine a predicted image of the building roof at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the predicted image of the roof of the tree occlusion image at the current time step; Determine the building roof occlusion image at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the tree occlusion image of the building roof to be restored; Based on the difference between the predicted image of the building roof at the occlusion position and the occlusion image of the building roof, a tree occlusion restoration change constraint from a current time step to a subsequent time step is determined.

[0014] According to the building roof tree occlusion restoration method provided by the present invention, the conditional diffusion restoration image of the current time step is obtained based on the restoration change constraint, and the conditional diffusion restoration image is fused and restored based on the tree occlusion fusion restoration sampler to obtain the tree occlusion image of the next time step, including: Based on the restoration change constraint, a conditional diffusion restoration image of the area blocked by trees at the current time step is obtained by a reverse sampling method of conditional control denoising; The diffusion image of the area not blocked by trees at the current time step is obtained by forward diffusion; Based on the tree occlusion fusion restoration sampler, the conditional diffusion restoration image is combined with the diffusion image and then fused and restored to obtain the tree occlusion image at the next time step.

[0015] The present invention also provides a tree-shading repair device for a building roof, comprising: An image acquisition unit, used for acquiring a tree-blocked image of the roof of the building to be repaired, wherein the tree-blocked image at the first time step is the tree-blocked image of the roof of the building to be repaired; An iterative denoising unit is used to repeatedly perform the following steps until a preset step length is reached: based on a tree occlusion decomposition model, predict an occlusion position image of the tree occlusion image at the current time step and a roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine a tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain a conditional diffusion repair image at the current time step, and based on a tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain a tree occlusion image at the next time step; The image determination unit is used to use the tree occlusion image at the last time step as the final restored complete building roof image.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned methods for repairing tree shading on the roof of a building is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for repairing tree shading on a building roof as described in any one of the above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for repairing tree shading on a building roof.

[0019] The method, device, electronic device and storage medium for repairing tree occlusion on the roof of a building provided by the present invention treat the tree occlusion image of the roof of the building to be repaired as a random noise image, gradually remove the noise pixels that occlude the roof, and restore the original image information of the roof. The characteristic distribution and change law of the pixels at the occlusion position when removing noise are learned through the tree occlusion decomposition model, and the final contour result is extracted after obtaining the target result of the roof repair image, thereby improving the accuracy of tree occlusion repair on the roof of the building. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is one of the flow charts of the method for repairing tree shading on the roof of a building provided by the present invention.

[0022] Figure 2 This is the second flow chart of the method for repairing tree shading on the roof of a building provided by the present invention.

[0023] Figure 3 Schematic diagram of a building roof example data set provided by the present invention.

[0024] Figure 4 It is a schematic diagram of a rooftop tree occlusion simulation sequence data set provided by the present invention.

[0025] Figure 5 It is one of the structural schematic diagrams of the tree occlusion decomposition model provided by the present invention.

[0026] Figure 6 This is the second structural schematic diagram of the tree occlusion decomposition model provided by the present invention.

[0027] Figure 7 This is the third flow chart of the method for repairing tree shading on the roof of a building provided by the present invention.

[0028] Figure 8 It is a structural schematic diagram of the tree-shading repairing device for the roof of a building provided by the present invention.

[0029] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In recent years, diffusion models have been widely used in natural images and remote sensing images. The tree occlusion factor in remote sensing images can be regarded as a kind of occlusion noise, which exists in the form of patches on the roof. Starting from the denoising idea, by gradually predicting the tree occlusion patches and the corresponding real roofs below them, it is used as the condition of the backward denoising process to guide the restoration of the complete building roof. The present invention proposes a method suitable for targeted restoration of the roof after obtaining the tree occlusion roof candidate frame or the preliminary extraction result, and then extracting the final contour result after obtaining the roof restoration image target result.

[0032] Figure 1 FIG. 1 is one of the flow charts of the method for repairing tree shading on the roof of a building provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining a tree occlusion image and a Gaussian pure noise image of the roof of the building to be repaired, wherein the tree occlusion image at the first time step is the tree occlusion image of the roof of the building to be repaired.

[0033] Specifically, the tree-occluded image of the building roof to be repaired is the building roof image that needs to be repaired with tree occlusion, that is, the tree-occluded image of the building roof includes tree occlusion noise. The repair process is the process of removing the tree occlusion noise and restoring the original image information of the roof. Tree occlusion noise refers to the phenomenon that the geometric and spectral characteristics of the roof are changed due to the occlusion of trees within the outline of the building roof in the remote sensing image.

[0034] Step 120, repeatedly perform the following steps until a preset step length is reached: based on the tree occlusion decomposition model, predict the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain the tree occlusion image of the next time step.

[0035] Step 130, using the tree occlusion image at the last time step as the final restored complete building roof image; Specifically, the embodiment of the present invention uses a T-step decomposition process based on the diffusion denoising concept to gradually remove tree occlusion noise from a tree occlusion image of a roof of a building to be restored until a roof restoration image is obtained. Figure 2 FIG. 2 is a flow chart of the method for repairing tree shading on a building roof provided by the present invention. Figure 2 As shown, for any time t, three modules are mainly processed: (1) Tree occlusion decomposition model: For the input image, the tree occlusion decomposition model is first used to obtain the current prediction of the occlusion position, that is, the occlusion position image of the tree occlusion image at the current time step is predicted, and the prediction of the original image, that is, the roof prediction image of the tree occlusion image at the current time step is predicted.

[0036] (2) Tree occlusion repair change constraints: After obtaining the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step, the tree occlusion repair change constraints from the current time step to the next time step can be determined based on the occlusion position image, the roof prediction image, and the tree occlusion image of the roof of the building to be repaired.

[0037] (3) Tree occlusion fusion restoration sampler: Based on the restoration change constraint, the generation direction of the occluded area is controlled to correspond to the distribution pattern of the visible area, thereby performing conditional control denoising on the Gaussian pure noise image and obtaining the conditional diffusion restoration image of the current time step. On this basis, based on the tree occlusion fusion restoration sampler, the conditional diffusion restoration image is fused and restored to obtain the denoising result of the current time step, that is, the tree occlusion image of the next time step.

[0038] Before executing step 120, a denoising diffusion model may be used to pre-train a complete building roof instance dataset, in order to understand the pixel distribution pattern of the building roof in a non-occluded state, including the geometric shape, spectral color, and texture style of each building roof instance.

[0039] The data sets of this embodiment include the building roof instance data set BUCEA4.0 and the roof tree occlusion simulation sequence data set. Figure 3 It is a schematic diagram of the building roof example dataset provided by the present invention. BUCEA4.0 defines the complete building roof repair standard, which is the key to the feasibility of repair. The diffusion model is pre-trained on this dataset to obtain the generation capability of the complete building roof.

[0040] The forward diffusion process is defined as follows: From the uniformly distributed building roof instance data Randomly sample a rooftop image sample ,right Add T times of Gaussian noise randomly sampled from a standard normal distribution Get the diffusion image . At the same time Input into the diffusion model neural network to learn the noise information added at each step and fit When T is large enough, the final noise image Will obey the standard normal distribution. According to the properties of the Markov chain, the forward noise enhancement process can obtain formula (1): in is the diffusion process image, is Gaussian noise, coefficient The value range is (0, 1) and satisfies formula (2): Then we have the following conditional probability distribution formula (3): in For known under conditions The conditional probability of . By recursion, we can directly get Calculated Formula (4): Depend on The obtained conditional probability distribution becomes formula (5): in For known under conditions The conditional probability of . Backward denoising is to gradually remove the noise from a pure noise image to obtain the generated image The process is as follows: First, a noise image is randomly sampled from a standard normal distribution. ,Will Adding noise to the deep neural network to get predictions , remove the noise to get the denoised image in the first step , repeat the above steps T times to get the final generated image According to the posterior probability distribution, we can get formula (6): in Indicates that in the known under conditions The posterior probability of . represents the parameters of the neural network. is the mean, is the variance.

[0041] In the original diffusion model, the variance is fixed and the neural network only calculates the mean. Make a prediction. Based on the noise prediction results We can get formula (7): The mean In a given Under the condition of only and current t Step diffusion process image , neural network parameters , and the predicted results related.

[0042] Based on the mathematical process of equations (1)-(7), the constructed complete building roof instance dataset is trained and self-supervised learning is used to obtain a pre-trained building roof diffusion model for subsequent restoration work. During the training process, the diffusion model learns the process of restoring the original image from different noise levels, understands the global structure and detail features of the image, and gradually understands the shape, texture and boundary information of the complete roof. Through the self-supervision mechanism, the model can extract features from the data itself without relying on explicit manual annotations. The pre-trained diffusion model can use the features learned from the complete building roof atlas sample library to generate realistic building roof images from pure noise images.

[0043] For the simulated sequence dataset of rooftop tree occlusion, it is usually difficult to determine the true boundary contour and true pixel information of the building roof and the background under the tree occlusion from a single data source in remote sensing image scenes. Therefore, there is a lack of label data corresponding to the occluded image and the real image for training. Compared with shadows, clouds, fog, etc., tree occlusion itself is a common feature type in high-resolution remote sensing images, and its pixels are concentrated and continuously distributed to form obvious tree occlusion spots.

[0044] Figure 4 is a schematic diagram of a rooftop tree occlusion simulation sequence data set provided by the present invention, such as Figure 4 As shown in the figure, an image of a building rooftop X (occluded image) blocked by trees can be regarded as a random spatiotemporal combination process of a complete building rooftop image B (clean image) and a tree image O. The position of tree occlusion is controlled by randomly generating a binary map P (position image, where the background is 0 and the tree is 1) of the occluded patch and multiplying it with the tree image O. At the same time, since there is an edge transition zone composed of mixed pixels at the edge of the tree, in order to make the simulated data closer to the actual environment, a Gaussian smoothing operation is performed on the edge of the occluded patch in P within a random width of 1-3 pixels. Therefore, the occluded image X can be expressed as the following formula (8): For deep learning models, the real image B is usually predicted directly from the occluded image X. Based on the analysis of shadow occlusion composition and inspired by the diffusion model, and in order to make the model's learning task fit the process of repairing tree occlusion, this simulation method decomposes formula (8) and defines it as the forward occlusion process of tree occlusion noise. The specific description is as follows: The tree occlusion patch noise is gradually added to the real image B, and the building roof and background pixels are gradually removed. The simulation process is specified as Step 1: Define a random noise list As shown in formula (9): The list elements The value is between 0 and 1, and the sum of all elements is 1. The default value is 5. At the i-th step, the occlusion position is proportional to Add a random tree occlusion image O to the image and remove the corresponding position of the real image B in equal proportion. Therefore, for any real image B, we can get formula (10): Formula (11) defines the tree occlusion noise added at each step : Formula (12) defines the real image to be removed : List The elements in are not completely random, but gradually increase according to the step size. Similar to the diffusion model, the number of trees added at the beginning is small, and the number of trees added later is large. At the beginning, the outline of the building roof below can be seen through the tree canopy, and the outline of the building roof becomes more and more blurred until it is completely blocked by the tree canopy. The sum of the sequence elements is always 1, ensuring that the final The output of the step , still satisfies formula (8).

[0045] In some embodiments, based on the tree occlusion decomposition model, an occlusion position image of the tree occlusion image at the current time step and a roof prediction image of the tree occlusion image at the current time step are predicted, that is, step 120 specifically includes: Step 121, based on the encoder, extracting the intermediate transition features of the tree occlusion image at the current time step; Step 122, predicting an occlusion position image of the tree occlusion image at the current time step based on the position decoder and the intermediate transition features; Step 123, based on the pixel decoder and the intermediate transition features, predict the roof prediction image of the tree occlusion image at the current time step.

[0046] Specifically, Figure 5 It is one of the structural schematic diagrams of the tree occlusion decomposition model provided by the present invention. The tree occlusion decomposition model includes an encoder, a position decoder and a pixel decoder. Figure 6 This is the second structural diagram of the tree occlusion decomposition model provided by the present invention. Figure 5 and Figure 6 As shown in Figure 1, the tree occlusion decomposition model is based on the network structure of the original diffusion model, which is constructed as a U-shaped structure with a single encoder and dual decoders (position decoder\pixel decoder). Its purpose is to simulate the sequence data of roof occlusion corresponding to any time i. After obtaining the image features through the four-layer encoder block, the tree occlusion position at the current time i is predicted from the two decoders after the middle transition layer of the Middle Block. and original pixels The decoder block has one more block per layer than the encoder block, which is used to receive the feature information of the short connection.

[0047] Encoder Block, Decoder-P Block and Decoder-B Block such as Figure 6As shown in (a). Their previous structures are the same, the difference is that the attention structure is optimized for different tasks of the encoder and decoder. The current time step i is superimposed with the input after group normalization, Swish activation function and Conv3×3 convolution through time embedding, and then after group normalization, Swish activation function, Conv3×3 convolution and Dropout, it is residually connected with the original input after Conv1×1. The Middle Block is composed of two Encoder Blocks connected for information transfer between the transition encoder and decoder. For the Encoder Block part, the most original self-attention is used to obtain the most complete overall image features.

[0048] It should be noted that step 122 and step 123 may be executed simultaneously or sequentially, and the execution order of the two is not limited.

[0049] Based on the pixel decoder and the intermediate transition features, the roof prediction image of the tree occlusion image at the current time step is predicted, including: Step 123-1, based on the pixel decoder, performing partial random masking on the intermediate transition features to obtain masked features and unmasked features; Step 123-2, perform convolution calculation on the masked features, perform multi-head self-attention calculation on the unmasked features, and fuse the calculation results to obtain the roof prediction image of the tree occlusion image at the current time step.

[0050] Specifically, Figure 6 As shown in (b), since the pixel decoder performs the task of raw pixel prediction, most pixels that are not affected by tree occlusion do not need to be changed. Therefore, a random masking method is used to reduce the complexity of the calculation. First, the input is divided along the spatial dimension, and then 50% is randomly masked. Only the unmasked 50% is subjected to multi-head self-attention operation, and the masked 50% is only passed through Conv3×3 convolution.

[0051] Based on the position decoder and the intermediate transition features, the occlusion position image of the tree occlusion image at the current time step is predicted, including: Step 122-1, calculating the attention score of the intermediate transition feature based on the position decoder; Step 122-2, perform multi-head self-attention calculation on the intermediate transition features with attention scores greater than or equal to the preset threshold, perform convolution calculation on the intermediate transition features with attention scores less than the preset threshold, and fuse the calculation results to obtain the occlusion position image of the tree occlusion image at the current time step.

[0052] Specifically, Figure 6As shown in (c), the position decoder actually performs a binary classification task, that is, the trees that block the roof are 1 and the background is 0. Self-attention calculation is performed only on those parts with higher feature values. First, the spatial dimension is split and the attention score of each part is calculated. After sorting, the top 50% scores are selected for multi-head self-attention module calculation and concat with the last 50% after Conv3×3 and then output.

[0053] Through the above-mentioned attention method, the attention to image features of the corresponding task can be improved and the computational complexity can be reduced to a certain extent.

[0054] The decomposition process of the tree occlusion decomposition model can redefine a random denoising list , satisfying the following formula (13): The list elements The value is between 0 and 1, and the sum of all elements is 1. The reason why they do not need to be completely consistent is that the model focuses on the trend of change from occlusion to unocclusion. The change of does not affect this trend, but only controls the change in the pixel value at the location where the tree occlusion occurs in each time step. By gradually removing the tree occlusion noise, the real image B can be restored to Formula (14) and Formula (15): in, and For the simulated image sequence Any The tree occlusion decomposition model consists of a shared image encoder and two feature decoders Decoder-P and Decoder-B, which respectively decode the occlusion position and original pixels Make a prediction and get After that, the tree occlusion noise is obtained directly according to the occlusion image to be repaired. The prediction formula (16): A random time step in the tree occlusion image sequence And the corresponding occlusion image , as input, enters the encoder to obtain the intermediate transition features Then They will be respectively entered into the position decoder Decoder-P for feature decoding to predict the occlusion position , that is, the occlusion position image of the tree occlusion image at the current time step is obtained; it enters the pixel decoder Decoder-B, the purpose of which is to decode and generate the real image , get pixel prediction , that is, the roof prediction image of the tree occlusion image at the current time step is obtained. As shown in formula (17): in, In the backward reasoning stage, for a real building roof image X that is blocked by trees, it is equivalent to the last element in the simulated data sequence. , that is, the tree occlusion is completely added, then after You will get , as shown in formula (18): And so on, after step length The image restoration result formula given by the decomposition model can be obtained (19): Based on the above embodiment, the loss function of the tree occlusion decomposition model is determined based on the positioning loss function, the prediction loss function and the restoration loss function; The positioning loss function is determined based on the occlusion position image of the tree occlusion image at the current time step and the sample true occlusion position image. The prediction loss function is determined based on the roof prediction image of the tree occlusion image at the current time step and the sample building roof image. The restoration loss function is determined based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step and the sample building roof image.

[0055] Specifically, the loss function of the tree occlusion decomposition model constructed in the embodiment of the present invention mainly includes the following three parts: Trees are widely distributed in remote sensing images, so it is critical to determine the specific location information of the tree occlusion. Position decoder Decoder-P can predict the positioning information by decoding the input image features. , we can calculate the prediction The L1 norm between the true value P is used to calculate the positioning loss, as shown in formula (20): In addition, the pixel decoder Decoder-B attempts to repair the roof of the building under the location and obtains the predicted building image result , we can calculate the prediction The restoration loss is calculated by the root mean square error between the true value B and the true value B, as shown in formula (21): Although the position decoder Decoder-P and the pixel decoder Decoder-B both obtain image feature information from the same encoder, they are still independent of each other during the decoding process. Therefore, the final output prediction result lacks the relationship between the occlusion position information and the corresponding tree information, building roof information, and background information. and , the predicted repair result is as follows formula (22): In order to evaluate the consistency between the restoration result and the real situation, the generated restoration image is made visually closer to the real image. The following formula (23) is shown: In summary, the overall loss function of the occlusion decomposition model is as follows (24): in , is the weight coefficient used to balance the impact of various losses. , The weights are set to 1, 2, and 1 respectively. represents the positioning loss function, represents the prediction loss function, represents the repair loss function.

[0056] Based on any of the above embodiments, Figure 7 FIG. 3 is a flow chart of the method for repairing tree shading on the roof of a building provided by the present invention. Figure 7 As shown, the present invention is based on the T-step decomposition process of the diffusion denoising idea, from a building roof occlusion image to be restored Gradually remove the occlusion noise until the roof restoration image is obtained . For any time t: The unoccluded area does not need to be processed, and its process image can be directly obtained using the forward diffusion noise enhancement method; while the occluded area starts from the randomly sampled Gaussian noise, and the diffusion conditions provided by the tree occlusion decomposition model are used to control the generation direction to make it correspond to the distribution pattern of the visible area; finally, the two parts are fused and sampled to complete the denoising result of the current step. .

[0057] In some embodiments, in step 120, based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determining the tree occlusion repair change constraint from the current time step to the next time step includes: Step 124, determining a predicted image of the building roof at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the predicted image of the roof of the tree occlusion image at the current time step; Step 125, determining the building roof occlusion image at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the tree occlusion image of the building roof to be restored; Step 126, based on the difference between the predicted image of the building roof at the occlusion position and the occlusion image of the building roof, determine the tree occlusion restoration change constraint from the current time step to the next time step.

[0058] Specifically, for the sampling process of the diffusion model, since the reverse denoising starts from a pure noise image, the step size T is usually very large, the generation process is slow and difficult to control, and it is impossible to accurately repair the occluded roof area. In view of this, the embodiment of the present invention is based on the conditional diffusion model, and the spatial and spectral domain distribution change gradient law of the pixels at the occluded position is used as a condition to control the generation process of the diffusion model. According to the gradient change law of the pixels in the occluded area and the characteristics of the roof itself in the unoccluded area, the occluded roof part is gradually repaired with the "denoising" process.

[0059] In order to synchronize the decomposition process of the tree occlusion model with the reverse sampling process of the diffusion model, the time steps of the two need to be unified first. The subsequent reverse sampling of the diffusion model in this embodiment uses the DDIM method, and the default denoising step size is 20.

[0060] For the task of repairing tree occlusion on the roof of a building, simply put, it is the process of restoring the tree pixels at the occlusion position to the building roof pixels. Through the tree occlusion decomposition model, we can The prediction of the occluded position is obtained from And the prediction of the real image , which is multiplied with the original image to obtain the prediction of the tree image at the occluded position , that is, multiply the occlusion position image of the tree occlusion image at the current time step by the roof prediction image of the tree occlusion image at the current time step to obtain the building roof prediction image at the occlusion position .Depend on arrive The transformation process is the task process of occlusion repair. According to the principle of the conditional diffusion model, the embodiment of the present invention converts and The gradient of the difference between is used as a conditional classifier to control the generation process of the diffusion model and guide the model to transform the pixel style of the repaired area. Therefore, the control condition can be obtained from formula (25): : in Represents the gradient of the occluded position compared to the entire image. is provided by the tree occlusion decomposition model, and its step length The default value is 5, which means that the inference process of the tree occlusion decomposition model is performed every 4 steps during the inpainting process of T=20 with reverse sampling, passing the control conditions of the inpainting.

[0061] Based on any of the above embodiments, in step 120, a conditional diffusion restoration image of the current time step is obtained based on the restoration change constraint, and a tree occlusion restoration image of the next time step is obtained after fusion restoration of the conditional diffusion restoration image based on the tree occlusion fusion restoration sampler, including: Step 127, based on the restoration change constraint, a conditional diffusion restoration image of the area blocked by trees at the current time step is obtained by a reverse sampling method of conditional control denoising; Step 128, obtaining a diffusion image of the area not blocked by trees at the current time step by forward diffusion; Step 129, based on the tree occlusion fusion restoration sampler, the conditional diffusion restoration image and the diffusion image are combined and then fused and restored to obtain a tree occlusion image at the next time step.

[0062] Specifically, for any tree-occluded image of the roof of a building to be restored, the parts outside the occluded position do not need to be processed and should be retained. The intermediate sampled image at any time step t, that is, the diffuse image of the area not blocked by trees at the current time step Formula (26) can be obtained using the following forward diffusion method: in, Represents the tree occlusion image of the building roof to be repaired, coefficient The value range is (0, 1). The area that needs to be repaired is unknown, so the intermediate image needs to be obtained by reverse sampling. , that is, the conditional diffusion repair image of the area blocked by trees at the current time step, according to the repair change constraint Then we can get formula (27): in, represents the diffusion image at any time t, represents the control condition, i.e., fix the change constraint. Based on the tree occlusion decomposition model, and The combination gives the conditional repair process image , as shown in formula (28): in, Represents the prediction of the occlusion position at any time i. It not only contains the visible position information of the roof, but also adds change constraints to the predicted occluded area to control the generation of the image. As shown in formula (29), the restoration result of the current time step is finally obtained through fusion sampling. .

[0063] in, Indicates known Down, The posterior probability of is the prediction of the repaired image at the next moment, that is, the tree occlusion image at the next time step.

[0064] The building roof obstruction repair device provided by the present invention is described below. The building roof obstruction repair device described below and the building roof tree obstruction repair method described above can be referenced to each other.

[0065] Figure 8 Schematic diagram of the structure of the tree-shading repair device for the building roof provided by the present invention. Figure 8 As shown, the device comprises: An image acquisition unit 810 is used to acquire a tree occlusion image and a Gaussian pure noise image of the roof of the building to be repaired, wherein the tree occlusion image at the first time step is the tree occlusion image of the roof of the building to be repaired; The iterative denoising unit 820 is used to repeatedly perform the following steps until a preset step length is reached: based on the tree occlusion decomposition model, predict the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain the tree occlusion image of the next time step; The image determination unit 830 is used to use the tree occlusion image at the last time step as the final restored complete building roof image.

[0066] The device provided by the embodiment of the present invention regards the occluded image to be repaired of the building roof as a random noise image, gradually removes the noise pixels that occlude the roof, and restores the original image information of the roof. The occlusion decomposition model is used to learn the characteristic distribution and change law of the pixels at the occluded position when removing the noise, and the target result of the roof repair image is obtained, and then the final contour result is extracted.

[0067] Based on any of the above embodiments, the iterative denoising unit is specifically used for: Based on the pixel decoder, the intermediate transition features are subjected to partial random masking to obtain masked features and unmasked features; A convolution calculation is performed on the masked features, a multi-head self-attention calculation is performed on the unmasked features, and the calculation results are fused to obtain a roof prediction image of the tree-occluded image at the current time step.

[0068] Based on any of the above embodiments, the loss function of the tree occlusion decomposition model is determined based on the positioning loss function, the prediction loss function and the restoration loss function; The positioning loss function is determined based on the occlusion position image of the tree occlusion image at the current time step and the sample true occlusion position image, the prediction loss function is determined based on the roof prediction image of the tree occlusion image at the current time step and the sample building roof image, and the restoration loss function is determined based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step and the sample building roof image.

[0069] Based on any of the above embodiments, the iterative denoising unit is specifically used for: Determine a predicted image of the building roof at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the predicted image of the roof of the tree occlusion image at the current time step; Determine the building roof occlusion image at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the tree occlusion image of the building roof to be restored; Based on the difference between the predicted image of the building roof at the occlusion position and the occlusion image of the building roof, a tree occlusion restoration change constraint from a current time step to a subsequent time step is determined.

[0070] Based on any of the above embodiments, the iterative denoising unit is specifically used for: Based on the restoration change constraint, a conditional diffusion restoration image of the area blocked by trees at the current time step is obtained by a reverse sampling method of conditional control denoising; The diffusion image of the area not blocked by trees at the current time step is obtained by forward diffusion; Based on the tree occlusion fusion restoration sampler, the conditional diffusion restoration image is combined with the diffusion image and then fused and restored to obtain the tree occlusion image at the next time step.

[0071] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the tree occlusion repair method for the building roof, which includes: obtaining a tree occlusion image and a Gaussian pure noise image of the building roof to be repaired, the tree occlusion image of the first time step is the tree occlusion image of the building roof to be repaired; repeating the following steps until a preset step length is reached: based on the tree occlusion decomposition model, predicting the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the building roof to be repaired, determining the tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally controlling the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fusing and repairing the conditional diffusion repair image to obtain the tree occlusion image of the next time step; taking the tree occlusion image of the last time step as the complete building roof image after the final repair.

[0072] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0073] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the tree occlusion repair method for the building roof provided by the above-mentioned methods. The method includes: obtaining a tree occlusion image and a Gaussian pure noise image of the building roof to be repaired, the tree occlusion image at the first time step is the tree occlusion image of the building roof to be repaired; repeatedly performing the following steps until a preset step length is reached: based on a tree occlusion decomposition model, predicting an occlusion position image of the tree occlusion image at the current time step, and the current time step A roof prediction image of an inter-time tree occlusion image; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraints from the current time step to the next time step; based on the repair change constraints, conditionally control denoising the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain the tree occlusion image of the next time step; use the tree occlusion image of the last time step as the final repaired complete building roof image.

[0074] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the tree occlusion repair method for the roof of a building provided by the above-mentioned methods, the method comprising: obtaining a tree occlusion image and a Gaussian pure noise image of the roof of the building to be repaired, the tree occlusion image at the first time step being the tree occlusion image of the roof of the building to be repaired; repeatedly performing the following steps until a preset step length is reached: based on a tree occlusion decomposition model, predicting an occlusion position image of the tree occlusion image at the current time step, and a roof prediction image of the tree occlusion image at the current time step ; Based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraints from the current time step to the next time step; based on the repair change constraints, conditionally control denoising is performed on the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, the conditional diffusion repair image is fused and repaired to obtain the tree occlusion image of the next time step; the tree occlusion image of the last time step is used as the complete building roof image after the final repair.

[0075] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for repairing tree shading on a building roof, characterized in that: include: Obtain a tree occlusion image and a Gaussian pure noise image of the roof of the building to be repaired, wherein the tree occlusion image at the first time step is the tree occlusion image of the roof of the building to be repaired; Repeat the following steps until a preset step length is reached: based on the tree occlusion decomposition model, predict the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine the tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain the conditional diffusion repair image of the current time step, and based on the tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain the tree occlusion image of the next time step; The tree occlusion image at the last time step is used as the final restored complete building roof image.

2. The method for repairing tree shading on a building roof according to claim 1, characterized in that: The tree occlusion decomposition model includes an encoder, a position decoder and a pixel decoder. The tree occlusion decomposition model is used to predict the occlusion position image of the tree occlusion image at the current time step and the roof prediction image of the tree occlusion image at the current time step, including: Based on the encoder, extracting intermediate transition features of the tree occlusion image at the current time step; Based on the position decoder and the intermediate transition features, predicting an occlusion position image of the tree occlusion image at the current time step; Based on the pixel decoder and the intermediate transition features, a roof prediction image of the tree occlusion image at the current time step is predicted.

3. The method for repairing tree shading on a building roof according to claim 2, characterized in that: The step of predicting the occlusion position image of the tree occlusion image at the current time step based on the position decoder and the intermediate transition feature comprises: Based on the position decoder, calculating an attention score of the intermediate transition feature; Multi-head self-attention calculation is performed on intermediate transition features whose attention scores are greater than or equal to a preset threshold, and convolution calculation is performed on intermediate transition features whose attention scores are less than the preset threshold. The calculation results are fused to obtain the occlusion position image of the tree occlusion image at the current time step.

4. The method for repairing tree shading on a building roof according to claim 2, characterized in that: The step of predicting a roof prediction image of the tree occlusion image at the current time step based on the pixel decoder and the intermediate transition feature comprises: Based on the pixel decoder, the intermediate transition features are subjected to partial random masking to obtain masked features and unmasked features; A convolution calculation is performed on the masked features, a multi-head self-attention calculation is performed on the unmasked features, and the calculation results are fused to obtain a roof prediction image of the tree-occluded image at the current time step.

5. The method for repairing tree shading on a building roof according to any one of claims 1 to 4, characterized in that: The loss function of the tree occlusion decomposition model is determined based on the positioning loss function, the prediction loss function and the restoration loss function; The positioning loss function is determined based on the occlusion position image of the tree occlusion image at the current time step and the sample true occlusion position image, the prediction loss function is determined based on the roof prediction image of the tree occlusion image at the current time step and the sample building roof image, and the restoration loss function is determined based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step and the sample building roof image.

6. The method for repairing tree shading on a building roof according to any one of claims 1 to 4, characterized in that: The determining of the tree occlusion restoration change constraint from the current time step to the next time step based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be restored comprises: Determine a predicted image of the building roof at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the predicted image of the roof of the tree occlusion image at the current time step; Determine the building roof occlusion image at the occlusion position based on the occlusion position image of the tree occlusion image at the current time step and the tree occlusion image of the building roof to be restored; Based on the difference between the predicted image of the building roof at the occlusion position and the occlusion image of the building roof, a tree occlusion restoration change constraint from a current time step to a subsequent time step is determined.

7. The method for repairing tree shading on a building roof according to any one of claims 1 to 4, characterized in that: The conditional diffusion restoration image of the current time step is obtained based on the restoration change constraint, and the tree occlusion restoration image of the next time step is obtained by fusing and restoring the conditional diffusion restoration image based on the tree occlusion fusion restoration sampler, including: Based on the restoration change constraint, a conditional diffusion restoration image of the area blocked by trees at the current time step is obtained by a reverse sampling method of conditional control denoising; The diffusion image of the area not blocked by trees at the current time step is obtained by forward diffusion; Based on the tree occlusion fusion restoration sampler, the conditional diffusion restoration image is combined with the diffusion image and then fused and restored to obtain the tree occlusion image at the next time step.

8. A tree-shading repair device for a building roof, characterized in that: include: An image acquisition unit, used to acquire a tree-blocked image and a Gaussian pure noise image of the roof of the building to be repaired, wherein the tree-blocked image at the first time step is the tree-blocked image of the roof of the building to be repaired; An iterative denoising unit is used to repeatedly perform the following steps until a preset step length is reached: based on a tree occlusion decomposition model, predict an occlusion position image of the tree occlusion image at the current time step and a roof prediction image of the tree occlusion image at the current time step; based on the occlusion position image of the tree occlusion image at the current time step, the roof prediction image of the tree occlusion image at the current time step, and the tree occlusion image of the roof of the building to be repaired, determine a tree occlusion repair change constraint from the current time step to the next time step; based on the repair change constraint, conditionally control denoising the Gaussian pure noise image to obtain a conditional diffusion repair image at the current time step, and based on a tree occlusion fusion repair sampler, fuse and repair the conditional diffusion repair image to obtain a tree occlusion image at the next time step; The image determination unit is used to use the tree occlusion image at the last time step as the final restored complete building roof image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for repairing tree occlusion on the roof of a building as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for repairing tree occlusion on a building roof as claimed in any one of claims 1 to 7 is implemented.

Images