Shielding facade sensing method and system based on feature distinguishing learning deep network

Abstract

The invention discloses an occlusion facade perception method and system based on a feature distinguishing learning deep network. The method comprises the steps of 1, acquiring an occlusion facade image set; step 2, inputting the occlusion facade image set into a deep network to obtain first-stage enhanced image deep representation; 3, respectively sending the deep representation of the image enhanced in the first stage into two auxiliary processing modules for processing to obtain two multi-scale enhanced representations with distinguishing hidden layer features; 4, obtaining a difference dichotomy semantic prediction map of the attention rule component and the attention background, and obtaining semantic prediction maps of all classes; and step 5, training a deep network based on the dichotomy semantic prediction map and all category semantic prediction maps, and storing the trained deep network. According to the method, the technical problem of how to effectively process shielding and analyze the facade image is solved, and the method has good application value in the fields of computer vision, graphics, facade image processing and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer graphics processing technology and computer vision, and in particular to a method and system for perceiving occluded facades based on a feature-differentiated learning deep network. Background Art

[0002] In the field of computer graphics and computer vision, facade parsing is an important issue of common concern, among which the parsing of occluded facade images is a very challenging derivative task. Its goal is to semantically segment occluded facade images and accurately identify and separate different components, such as windows, doors, and balconies. This technology is of great significance in practical applications, such as 3D reconstruction of buildings, autonomous driving, and urban planning.

[0003] In early work, traditional methods such as Teboul et al. (Segmentation of building facades using procedural shape priors, 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, 2010, pp. 3105–3112) decomposed facade images into various categories and further parsed them into basic elements using predefined grammar derivation trees and random forest classifiers, making significant contributions to facade parsing; Jiang et al. (Automatic constraint detection for 2D layout regularization, IEEE transactions on visualization and computer graphics 22 (8) (2015) 1933–1944) introduced the concept of layout regularization into semantic segmentation of appearance parsing, and improved the accuracy and efficiency of semantic segmentation by automatically detecting layout constraints and optimizing input layouts, making the parsing results more in line with architectural design standards; Liu et al. (A kronecker product model for repeated pattern detection on 2D urban images, IEEE transactions on pattern analysis and machine intelligence 41 (9) (2018) 22662272) combined manually designed feature extraction with the Kronecker product model to detect repetitive patterns of building components based on building facades. However, traditional methods usually rely on static rule definitions and hand-crafted feature representations, which show limitations when dealing with facade images containing contaminated or missing parts. In addition, relying solely on rule-based priors can cause the arrangement of components to become monotonous, making it difficult to accurately recreate the original facade layout pattern. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a method and system for perceiving occluded facades based on a feature-differentiated learning deep network, which achieves high-quality semantic segmentation of occluded facade images and reduces the semantic ambiguity of regular and irregular features in the network latent space.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for perceiving an obstructed facade based on a feature-differentiated learning deep network, comprising the following steps: Step 1: Obtain a set of occluded facade images as the input source of the deep network; Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement. Step 3: Send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features; Step 4: The first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features are processed by the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the attention rule component and a binary semantic prediction map of the attention background; the deep representation of the image enhanced in the first stage and the first multi-scale enhanced representation and the second multi-scale enhanced representation are superimposed and processed by the main decoder to obtain semantic prediction maps of all categories; Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0006] On the other hand, the present invention also provides an occluded facade perception system based on a feature-differentiated learning deep network, the system comprising: An input source acquisition module is used to obtain a set of occluded facade images as an input source for the deep network; The image deep representation module is used to input the occluded facade image set into the deep network. The deep network predicts the occluded area in combination with the contextual semantic environment, and obtains the first-stage enhanced image deep representation after the mainstream convolution processing module; A multi-scale enhanced representation module, used to send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, so as to obtain a first multi-scale enhanced representation and a second multi-scale enhanced representation with distinguishing hidden layer features; A semantic prediction map acquisition module is used to process the first multi-scale enhanced representation and the second multi-scale enhanced representation through the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the focus rule component and a binary semantic prediction map of the focus background; superimpose the deep representation of the image enhanced in the first stage and the first multi-scale enhanced representation and the second multi-scale enhanced representation, and process them through the main decoder to obtain semantic prediction maps of all categories; The training module is used to train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0007] In a third aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0008] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0009] The beneficial effects of the present invention are: The present invention proposes a novel facade image parsing framework, which adopts a strong supervised training mode to distinguish different components when training the neural network. Compared with the traditional convolution-based facade parsing network, the facade image parsing framework of the present invention can more accurately predict the component shape and arrangement of the occluded area, solves the problem of how to predict the digital image of the facade that is occluded or affected by noise with high quality, and reduces the possible misprediction during the parsing process.

[0010] In addition, the present invention adopts a discriminative loss function to train the entire network, so that the facade image parsing framework can effectively balance the optimization problem of distinguishing regular categories and background categories, significantly reducing the training difficulty brought by traditional methods, and has broad application value in the fields of facade parsing and semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of an occluded facade perception method based on a feature-differentiated learning deep network of the present invention; Figure 2 It is the artificially synthesized occlusion elevation diagram of the present invention; Figure 3 This is a schematic diagram of a method for perceiving an obstructed facade based on a feature-differentiated learning deep network according to the present invention; Figure 4 This is an implementation effect diagram of an occluded facade perception method based on a feature-differentiated learning deep network of the present invention; Figure 5 This is a structural block diagram of an occluded facade perception system based on a feature-differentiated learning deep network according to the present invention. DETAILED DESCRIPTION

[0012] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0013] It should be noted that, in the absence of clear limitation or conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other. The present invention will be further described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the protection scope of the present application.

[0014] The core idea of ​​the present invention is to fully consider the shape distinction and position arrangement distinction of the features of different categories of components of the facade at different scales. In the training process of the encoding-decoding convolutional network, long-short distance feature processing units are used to perform fine-grained and coarse-grained multi-scale training on the features. This training method makes it possible to more effectively reduce feature confusion by distinguishing between regular categories and background categories, so that more reasonable global predictions and occluded area predictions can be obtained in the final output. This method not only improves the accuracy of the prediction, but also enhances the robustness of the model, and has significant advantages in dealing with complex facade image parsing problems.

[0015] The main technical concept of the present invention is to use an encoder and a decoder to map an image to a latent feature space, or to convert from a latent feature space to a semantic image space. In the latent feature space, the mainstream processing module and the auxiliary stream processing module are used to enhance the semantic structural information of the occluded image in a convolutional manner. The facade parsing network and loss function proposed in the present invention treat different features discriminatively, thereby having a significant effect in eliminating feature confusion. The facade parsing network does not require any hyperparameters to adjust the optimization target, making the training process simpler and more efficient. Most of the existing deep facade image parsing work only uses generators and decoders for conventional information enhancement, and does not take into account the differences between features and how to amplify such differences during training. The present invention successfully applies the framework and loss function of enhancing feature discrimination to the facade image parsing problem, thereby effectively filling in the missing or occluded parts and improving the accuracy and reliability of the prediction.

[0016] The embodiment of the present invention provides a method for perceiving an obstructed facade based on a feature-differentiated learning deep network. Figure 1 As shown, the method may include the following steps: Step 1: Obtain a set of occluded facade images as the input source of the deep network.

[0017] like Figure 2As shown in the figure, the occluded image and the true semantic image need to be paired in the training stage, but it is often difficult to collect paired images in real scenes. Therefore, the training data of the present invention is to remove a certain area of ​​the original facade image according to the occlusion mask on the clean facade image, and then create a natural occlusion in this area according to the collected tree and leaf images, so as to obtain an artificially synthesized occluded facade image paired with the true semantic image and the mask. The above process can be described as , where the clean facade image is represented as , the occluded facade image is represented as , the occlusion mask is expressed as (0 indicates valid area, 1 indicates missing area), is the pixel-by-pixel multiplication, Indicates the number of leaves currently selected.

[0018] Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement. In this stage, the deep representation of the image after the first stage enhancement is further optimized in terms of fine-grained representation.

[0019] like Figure 3 As shown, step 2 may specifically include: Step 2.1: The deep network uses an encoder composed of five layers of dilated fully convolutional layers FCN to project the input occluded facade image into the latent feature space, thereby obtaining a deep representation of the occluded facade image. The size of the convolution kernel of the first layer network is , the step size is 2; the convolution kernel sizes of the second and third layers are , the step size is 2, the expansion ratio of the first three layers is 1; the convolution kernel size of the fourth and fifth layers is , the step length is 1, and the expansion ratio is 2. Therefore, the size of the original input occluded facade image is downsampled twice from becomes deep representation of .

[0020] Step 2.2: In the latent feature space, the deep representation of the occluded facade image is subjected to preliminary feature comprehensive analysis by the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

[0021] The mainstream convolution processing module includes a LSCA-M convolution processing module. The LSCA-M convolution processing module is composed of a short-distance convolution processing branch and a long-distance convolution processing branch in parallel; The short-distance convolution processing branch includes a size of The horizontal one-dimensional convolution of size is A vertical one-dimensional convolution of size Square convolution Three parallel convolutions; The long-distance convolution processing branch consists of (expansion rate 3, size A horizontal strip convolution of size The combined convolution blocks of vertical strip convolutions in series to achieve a processing effect similar to horizontal rectangular convolutions) and (dilation rate is 3, size is The vertical strip convolution of size is The method uses convolution processing modes of multiple scales and sizes to fully integrate and consider the feature differences in the image, thereby improving the accuracy of image classification.

[0022] Step 3: Send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with distinguishing hidden layer features; specifically including: The deep representation of the image enhanced by the mainstream convolution processing module (LSCA-M) is respectively guided by two auxiliary modules ( Figure 3 The first auxiliary processing module and the second auxiliary processing module shown in the figure are further multi-scale represented. Among them, both auxiliary processing modules are composed of an average pooling layer with a step size of 2, an LSCA-S module and a linear interpolation upsampling layer. The LSCA-S module is a variant of the LSCA-M module. Since it processes on a smaller feature map, only the kernel size has changed and some redundant operations have been deleted. The LSCA-S module also includes a short-distance processing branch and a long-distance processing branch. Specifically, the short-distance processing branch of the LSCA-S module includes a horizontal one-dimensional convolution with a specific parameter size, a vertical one-dimensional convolution with a specific parameter size and a Convolution; The long-distance processing branch of the LSCA-S module contains approximately horizontal rectangular convolutions with a specific size and a dilation rate of 3, and approximately vertical rectangular convolutions with a specific size and a dilation rate of 3 connected in parallel.

[0023] Step 4: The first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features are processed by the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the focus rule component and a binary semantic prediction map of the focus background; the deep representation of the image enhanced in the first stage and the first multi-scale enhanced representation and the second multi-scale enhanced representation are superimposed and processed by the main decoder to obtain semantic prediction maps of all categories.

[0024] Step 4 specifically includes: Step S4.1: The upsampling process of two auxiliary decoders with different outputs and the same structure includes a convolution with a convolution kernel size of 3×3 and a bilinear interpolation upsampling layer with a 2-fold expansion, which includes a total of 3 such upsampling processes. The first time, the feature map of 16×16×512 is upsampled to a feature map of 32×32×256, the second time, the feature map of 32×32×256 is upsampled to a feature map of 64×64×128, and the third time, the feature map of 64×64×128 is upsampled to a feature map of 128×128×64. Finally, a linear layer and a softmax activation function are used to output the 128×128×64 feature map as a 128×128×2 semantic label prediction map. The output of the first auxiliary processing module is output through the first auxiliary decoder. The binary classification prediction map pays more attention to the set rule category, and the output of the second auxiliary processing module pays more attention to other categories (background) after passing through the second auxiliary decoder. The two binary classification semantic prediction maps with different focus categories output by the two auxiliary decoders will be used as the object of calculating the distinction loss in step 5.

[0025] Step 4.2: The upsampling process of the main decoder includes a convolution with a kernel size of 3×3 and a bilinear interpolation upsampling layer with a 2x dilation. The upsampling process is performed three times. The first time, the 32×32×512 feature map is upsampled to a 64×64×256 feature map, the second time, the 64×64×256 feature map is upsampled to a 128×128×128 feature map, and the third time, the 128×128×128 feature map is upsampled to a 256×256×64 feature map. Finally, a linear layer and a softmax activation function are used to output the 256×256×64 feature map as a 256×256×numclass semantic label prediction map, where numclass is the number of categories of all components in the facade image. The output of the main decoder is the final predicted label of all categories, which is the object of calculating the multi-class cross entropy loss in step 5.

[0026] Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0027] Step 5 may specifically include: Step 5.1: The binary semantic prediction maps output by the two auxiliary decoders are guided by the discriminative loss function. Specifically, the first auxiliary processing module focuses on a specific category, such as windows. When calculating the cross entropy loss of the binary classification, a larger weight is given to the pixel area whose true value is a window. This can help the model better identify and understand this specific category, thereby improving the model's prediction accuracy for this category. At the same time, the second auxiliary processing module focuses on other categories. When calculating the cross entropy loss of the binary classification, a larger weight is given to the pixel area whose true value is not a window. The binary cross loss calculated by the output of the two auxiliary decoders is added together to obtain the discriminative loss, which can help the model better identify and understand the non-window categories, thereby improving the model's prediction accuracy for these categories.

[0028] , Where N is the total number of input samples. is the true binary classification label value. Represents the predicted probabilities of different classes by the network.

[0029] Step 5.2: The output semantic prediction maps of all categories are guided using the multi-classification cross loss function.

[0030] , Where N is the total number of input samples and C is the total number of categories for multi-classification prediction. is the true multi-classification label value, Represents the predicted probabilities of different classes by the network.

[0031] Finally, the trained network can accept a facade image that may or may not contain occlusions as input. In order to optimize memory usage and increase prediction speed, the auxiliary decoder is only used during the training phase to guide the training of the network. In the actual prediction process, these two auxiliary decoders and their outputs are not needed. The main decoder directly generates a multi-classification semantic label prediction map, which simplifies the prediction process.

[0032] In summary, the present invention parses occluded facade images through a convolution-based image segmentation network, which has important application value in the fields of image segmentation and facade analysis.

[0033] In this embodiment, each step is described in the above-mentioned order. Those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously or in a reversed order. These simple changes are within the scope of protection of the present invention.

[0034] like Figure 4As shown in the figure, the facade analysis and prediction effects of the deep network occluded facade perception method based on feature discrimination learning proposed in the present invention on three different datasets ECP, Ruemonge2014 and CMP. In the figure, the first row shows the input facade image, the second row presents the corresponding real multi-classification semantic label, and the third row shows the predicted label map of the method of the present invention. Observing the first three columns, these input images are facade images without occlusion. It can be found that the prediction results of the present invention are basically consistent with the real labels and have high accuracy. The middle three columns show facade images in low-light scenes. Although the ambient light is weak, the method of the present invention can still make accurate predictions by utilizing powerful long-range and short-range contextual feature representations. In the images in the last three columns, there are a large number of categories, but even in the face of complex situations, the method of the present invention still shows excellent prediction performance.

[0035] like Figure 5 As shown, based on the same technical concept as the method embodiment, a system for sensing an occluded facade based on a feature-differentiated learning deep network is also provided. The system comprises at least: an input source acquisition module and a parsing network module. Among them, the acquisition module is configured to acquire an occluded facade image. The parsing network module is configured to map the occluded facade image to an implicit feature space. In this implicit feature space, based on the semantic information of the training image set and the surrounding unobstructed area, the module can perform a deep analysis of the occluded image block through a convolution operation. This parsing process is based on prior knowledge, that is, the regular arrangement and shape of the existing facade components, so that the image blocks can be reorganized in a structured manner. Through this parsing and reorganization operation, a final semantic prediction image can be obtained, which has high accuracy and credibility for the prediction of the occluded area. The training module includes a distinguishing loss function submodule and a multi-category cross entropy loss function submodule for network training.

[0036] In an optional embodiment, the parsing network module specifically includes: an image deep representation submodule, a multi-scale enhanced representation submodule, and a semantic prediction map acquisition submodule. Among them, the image deep representation submodule is configured to downsample the damaged image and obtain a deep representation in the latent feature space. The multi-scale enhanced representation submodule is configured to perform preliminary feature shape and rule analysis in the latent feature space, and perform certain repairs on the occluded area to obtain an enhanced multi-scale facade feature representation. The semantic prediction map acquisition upsamples the different feature maps of the output to obtain a distinguishing semantic map with different emphases on the rule category of concern and the background, and a multi-classification semantic map with prediction labels for all categories.

[0037] In an optional embodiment, the training module specifically includes: a discriminative loss function submodule and a multi-category cross entropy loss function submodule. Among them, the discriminative loss function submodule is configured to assign different attention components to the semantic graphs upsampled after processing the two auxiliary streams. Taking the regular category window as an example, the two-category semantic graph output by auxiliary stream 1 gives a greater weight to the pixel area whose true value is the window when calculating the cross entropy loss of the two categories. The two-category semantic graph output by auxiliary stream 2 gives a greater weight to the pixel area whose true value is not the window when calculating the cross entropy loss of the two categories. The multi-category cross entropy loss function submodule is configured to calculate the multi-category cross loss of the entire semantic prediction graph and the true value label in a supervised learning manner by using the semantic graph of all component categories that are finally output.

[0038] The present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned occluded facade perception method based on feature-differentiated learning deep network.

[0039] The present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0040] It should be noted that: the occluded facade perception system based on feature discrimination learning deep network provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing semantic prediction on occluded images. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0041] The above-mentioned system embodiment can be used to execute the above-mentioned method embodiment. Its technical principles, technical problems solved and technical effects produced are similar. Technical personnel in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0042] It should be noted that the system embodiment and method embodiment of the present invention are described above respectively, but the details described in one embodiment can also be applied to another embodiment. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and are not regarded as improper limitations of the present invention. Those skilled in the art should understand that the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0043] The technical solutions provided by the embodiments of the present invention are described in detail above. Although this article uses specific examples to illustrate the principles and implementation methods of the present invention, the description of the above embodiments is only applicable to help understand the principles of the embodiments of the present invention; at the same time, for those skilled in the art, changes will be made in the specific implementation methods and application scopes according to the embodiments of the present invention.

[0044] It should be noted that the marks and texts in the accompanying drawings are only for more clear explanation of the present invention and shall not be regarded as improper limitation of the protection scope of the present invention.

[0045] The terms "include", "comprises" or any other similar terms are intended to cover non-exclusive inclusion, so that a process, method, article or device / apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to these processes, methods, articles or devices / apparatuses. In the absence of further restrictions, the elements defined by the sentence "comprising one..." do not exclude the presence of additional elements in the process, method, article or device / apparatus comprising the elements, that is, the meaning of "comprising one" also covers the meaning of "comprising another".

[0046] The various steps of the present invention can be implemented by a general-purpose computing device, for example, they can be concentrated on a single computing device, such as a personal computer, a server computer, a handheld device or a portable device, a tablet device or a multi-processor device, or distributed on a network composed of multiple computing devices, which can perform the steps shown or described in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific hardware and software or their combination.

[0047] The method provided by the present invention can be implemented using a programmable logic device, or can be implemented as a computer program software or a program module (which includes routines, programs, objects, components or data structures, etc. that perform specific tasks or implement specific abstract data types). For example, according to an embodiment of the present invention, it can be a computer program product, and running the computer program product enables the computer to execute the method for demonstration. The computer program product includes a computer-readable storage medium, which contains a computer program logic or code portion for implementing the method. The computer-readable storage medium can be a built-in medium installed in a computer or a removable medium that can be removed from the computer body (for example, a storage device using hot-swap technology). The built-in medium includes, but is not limited to, a rewritable non-volatile memory, such as RAM, ROM, flash memory and hard disk. The removable medium includes, but is not limited to, optical storage media (for example, CD-ROM and DVD), magneto-optical storage media (for example, MO), magnetic storage media (for example, magnetic tape or mobile hard disk), a medium with a built-in rewritable non-volatile memory (for example, a memory card) and a medium with a built-in ROM (for example, a ROM box).

[0048] The present invention is not limited to the above-mentioned embodiments. Any modification, improvement or substitution that can be conceived by a person skilled in the art without departing from the essential content of the present invention shall fall within the scope of the present invention.

[0049] While the foregoing has shown, described and pointed out in detail the basic novel features of the present invention as applied to various embodiments, it will be understood that various omissions, substitutions and changes in the form and details of the system may be made by those skilled in the art without departing from the intent of the invention.

Claims

1. A method for occluded facade perception based on feature-differentiated learning deep network, characterized in that: The method comprises: Step 1: Obtain a set of occluded facade images as the input source of the deep network; Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement. Step 3: Send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features; Step 4: The first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features are processed by the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the attention rule component and a binary semantic prediction map of the attention background; the deep representation of the image enhanced in the first stage and the first multi-scale enhanced representation and the second multi-scale enhanced representation are superimposed and processed by the main decoder to obtain semantic prediction maps of all categories; Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

2. According to claim 1, a method for sensing obstructed facades based on feature-differentiated learning deep networks is characterized in that: The step 1 specifically includes: The artificially synthesized occluded facade image is determined according to the following formula: ， in, represents the occluded facade image, Represents a clean facade image, represents the occlusion mask, represents pixel-by-pixel multiplication, Indicates the number of leaves currently selected.

3. According to claim 1, the method for sensing obstructed facades based on feature-differentiated learning deep networks is characterized in that: The step 2 specifically includes: Step 2.1, the deep network uses an encoder composed of five dilated full convolutional layers to project the input occluded facade image into the latent feature space to obtain a deep representation of the occluded facade image; Step 2.2: In the latent feature space, the deep representation of the occluded facade image is subjected to preliminary feature comprehensive analysis by the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

4. The method for sensing obstructed facades based on feature-differentiated learning deep networks according to claim 3 is characterized in that: The mainstream convolution processing module includes an LSCA-M submodule that performs comprehensive feature understanding at short and long distances on the deep representation of the image downsampled by the encoder.

5. According to claim 1, the method for sensing obstructed facades based on feature-differentiated learning deep networks is characterized in that: The first auxiliary processing module and the second auxiliary processing module have the same structure, both of which include a pooling layer submodule for reducing the size of the feature map, an LSCA-S submodule, and a bilinear upsampling layer submodule for increasing the size of the feature map.

6. The method for sensing obstructed facades based on feature-differentiated learning deep networks according to claim 1, characterized in that: The main decoder includes: a bilinear upsampling layer submodule that increases the size of the feature map.

7. The method for sensing obstructed facades based on feature-differentiated learning deep networks according to claim 1, characterized in that: The step 5 specifically includes: Step 5.1: The binary semantic prediction maps output by the two auxiliary decoders are guided using the discriminative loss function; Step 5.2: All category semantic prediction maps are guided using the multi-class cross loss function.

8. An occluded facade perception system based on feature-differentiated learning deep network, characterized in that: The system comprises: An input source acquisition module is used to obtain a set of occluded facade images as an input source for the deep network; The image deep representation module is used to input the occluded facade image set into the deep network. The deep network predicts the occluded area in combination with the contextual semantic environment, and obtains the first-stage enhanced image deep representation after the mainstream convolution processing module; A multi-scale enhanced representation module, used to send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, so as to obtain a first multi-scale enhanced representation and a second multi-scale enhanced representation with distinguishing hidden layer features; A semantic prediction map acquisition module is used to process the first multi-scale enhanced representation and the second multi-scale enhanced representation through the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the focus rule component and a binary semantic prediction map of the focus background; superimpose the deep representation of the image enhanced in the first stage and the first multi-scale enhanced representation and the second multi-scale enhanced representation, and process them through the main decoder to obtain semantic prediction maps of all categories; The training module is used to train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Among them, when one or more programs are executed by the one or more processors, the one or more processors implement the occluded facade perception method based on feature-differentiated learning deep network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement a method for perceiving an obstructed facade based on a feature-differentiated learning deep network as described in any one of claims 1 to 7.

Images