Occluded facade perception method and system based on feature-differentiated learning deep network

Abstract

The application discloses a kind of based on feature distinguishing learning deep network's occlusion facade perception method and system, the method includes step 1: obtaining occlusion facade image set;Step 2: the occlusion facade image set is input into depth network, and the deep layer representation of first stage enhanced image is obtained;Step 3: the deep layer representation of first stage enhanced image is sent into two auxiliary processing modules respectively and is handled, obtains two multi-scale enhanced representations with distinguishing hidden layer features;Step 4: the differential binary classification semantic prediction graph of attention rule component and attention background is obtained, and all category semantic prediction graph is obtained;Step 5: based on binary classification semantic prediction graph and all category semantic prediction graph, train depth network and save the depth network of well-trained.The application solves the technical problem of how to effectively process occlusion and parse facade image, and has good application value in the field of computer vision, graphics and facade image processing.

Description

Technical Field

[0001] The present invention belongs to the fields of computer graphics processing technology and computer vision, and in particular relates to a method and system for perceiving occluded facades based on a feature-differentiated learning deep network. Background Art

[0002] Facade parsing is a critical problem in both computer graphics and computer vision. Parsing occluded facade images is a particularly challenging derivative task. Its goal is to perform semantic segmentation on occluded facade images, accurately identifying and separating distinct components such as windows, doors, and balconies. This technology is crucial for real-world applications such as 3D building reconstruction, autonomous driving, and urban planning.

[0003] In early work, traditional methods such as Teboul et al. (Segmentation of building facades using procedural shape priors, 2010 IEEE computer society conference on computer vision and pattern recognition, IEEE, 2010, pp. 3105–3112) decomposed facade images into various categories and further parsed them into basic elements using predefined grammar-derived trees and random forest classifiers, making significant contributions to facade parsing; Jiang et al. (Automatic constraint detection for 2d layout regularization, IEEE transactions on visualization and computergraphics 22 (8) (2015) 1933–1944) introduced the concept of layout regularization into semantic segmentation for appearance parsing, and improved the accuracy and efficiency of semantic segmentation by automatically detecting layout constraints and optimizing input layouts, making the parsing results more in line with architectural design standards; Liu et al. (A kronecker product model for repeated pattern detection on 2d urban images, IEEE transactions on pattern analysis and machine intelligence 41 (9) (2018) 22662272) combined manually designed feature extraction with the Kronecker product model to detect recurring patterns in building components based on building facades. However, traditional methods often rely on static rule definitions and handcrafted feature representations, which exhibit limitations when dealing with facade images containing contaminated or missing parts. Furthermore, relying solely on rule-based priors results in monotonous component arrangements, making it difficult to accurately recreate the original facade layout pattern. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a method and system for occluded facade perception based on a feature-discriminative learning deep network, which realizes high-quality semantic segmentation of occluded facade images and reduces the semantic ambiguity of regular and irregular features in the network latent space.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for perceiving an obstructed facade based on a feature-differentiated learning deep network, comprising the following steps:

[0007] Step 1: Obtain a set of occluded facade images as the input source of the deep network;

[0008] Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

[0009] Step 3: The deep representation of the image after the first stage enhancement is sent to the first auxiliary processing module and the second auxiliary processing module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features;

[0010] Step 4: The first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features are processed by the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the attention rule component and a binary semantic prediction map of the attention background; the deep representation of the image enhanced in the first stage is superimposed with the first multi-scale enhanced representation and the second multi-scale enhanced representation, and processed by the main decoder to obtain a semantic prediction map of all categories;

[0011] Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0012] On the other hand, the present invention also provides an occluded facade perception system based on a feature-differentiated learning deep network, the system comprising:

[0013] The input source acquisition module is used to obtain a set of occluded facade images as the input source of the deep network;

[0014] The deep image representation module is used to input the set of occluded facade images into a deep network. The deep network predicts the occluded area based on the contextual semantic environment and obtains the first-stage enhanced deep image representation after the mainstream convolution processing module.

[0015] A multi-scale enhanced representation module is used to send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, thereby obtaining a first multi-scale enhanced representation and a second multi-scale enhanced representation with discriminative hidden layer features;

[0016] The semantic prediction map acquisition module is used to process the first multi-scale enhanced representation and the second multi-scale enhanced representation through the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the focus rule component and a binary semantic prediction map of the focus background; the deep representation of the image enhanced in the first stage is superimposed with the first multi-scale enhanced representation and the second multi-scale enhanced representation, and processed through the main decoder to obtain semantic prediction maps of all categories;

[0017] The training module is used to train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0018] In a third aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0020] The beneficial effects of the present invention are:

[0021] This paper proposes a novel facade image parsing framework that employs a strongly supervised training model that distinguishes between different components when training a neural network. Compared to traditional convolution-based facade parsing networks, this framework can more accurately predict the shape and arrangement of components in occluded areas. This solves the problem of high-quality prediction of digital facade images that are occluded or affected by noise, while also reducing the potential for mispredictions during the parsing process.

[0022] In addition, the present invention uses a discriminative loss function to train the entire network, so that the facade image parsing framework can effectively balance the optimization problem of distinguishing regular categories and background categories, significantly reducing the training difficulty brought by traditional methods, and has wide application value in fields such as facade parsing and semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of an occluded facade perception method based on a feature-differentiated learning deep network according to the present invention;

[0024] Figure 2 It is the artificially synthesized occlusion elevation diagram of the present invention;

[0025] Figure 3This is a schematic diagram of a method for perceiving occluded facades based on a feature-differentiated learning deep network according to the present invention;

[0026] Figure 4 This is an implementation effect diagram of an occluded facade perception method based on a feature-differentiated learning deep network of the present invention;

[0027] Figure 5 This is a structural block diagram of an occluded facade perception system based on a feature-differentiated learning deep network in the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples and the accompanying drawings.

[0029] It should be noted that, unless expressly limited or conflicting, the embodiments of the present application and the technical features in the embodiments can be combined with each other. The present invention will be further described in detail below with reference to the accompanying drawings and in conjunction with the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work should fall within the scope of protection of the present application.

[0030] The core concept of this invention is to fully consider the shape and positional arrangement of features of different types of facade components at different scales. During the training process of the encoder-decoder convolutional network, long- and short-range feature processing units are used to perform fine-grained and coarse-grained multi-scale training of features. This training method makes it possible to more effectively reduce feature confusion when distinguishing between regular and background categories, thereby obtaining more reasonable global predictions and occlusion area predictions in the final output. This method not only improves the accuracy of predictions but also enhances the robustness of the model, which has significant advantages for handling complex facade image parsing problems.

[0031] The main technical concept of the present invention is to use an encoder and a decoder to map an image to a latent feature space, or to convert from a latent feature space to a semantic image space. In the latent feature space, the mainstream processing module and the auxiliary stream processing module are used to enhance the semantic structural information of the occluded image in a convolutional manner. The facade parsing network and loss function proposed in the present invention treat different features discriminatively, thereby having a significant effect in eliminating feature confusion. The facade parsing network does not require any hyperparameters to adjust the optimization target, making the training process simpler and more efficient. Most of the existing deep facade image parsing work only uses generators and decoders for conventional information enhancement, and does not take into account the differences between features and how to amplify such differences during training. The present invention successfully applies the framework and loss function for enhancing feature discrimination to the facade image parsing problem, thereby effectively filling in the missing or occluded parts and improving the accuracy and reliability of the prediction.

[0032] The embodiment of the present invention provides a method for perceiving occluded facades based on a feature-based learning deep network. Figure 1 As shown, the method may include the following steps:

[0033] Step 1: Obtain a set of occluded facade images as the input source of the deep network.

[0034] like Figure 2 As shown in the figure, the occlusion image and the ground truth semantic image need to be paired during the training phase, but it is often difficult to collect paired images in real scenes. Therefore, the training data of the present invention is to remove a certain area of ​​the original facade image based on the occlusion mask on the clean facade image, and then create a natural occlusion in this area based on the collected tree and leaf images, so as to obtain an artificially synthesized occlusion facade image paired with the ground truth semantic image and the mask. The above process can be described as ,where the clean facade image is represented as , the occluded facade image is represented as , the occlusion mask is expressed as (0 indicates valid area, 1 indicates missing area), is pixel-by-pixel multiplication, Indicates the original image of the currently selected number of leaves.

[0035] Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

[0036] In this stage, the deep representation of the image after the first stage enhancement is further optimized in terms of fine-grained representation.

[0037] like Figure 3As shown, step 2 may specifically include:

[0038] Step 2.1: The deep network uses an encoder composed of five layers of dilated fully convolutional layers (FCN) to project the input occluded facade image into the latent feature space, thereby obtaining a deep representation of the occluded facade image. The convolution kernel size of the first layer is , the step size is 2; the convolution kernel sizes of the second and third layers are , the step size is 2, the expansion ratio of the first three layers is 1; the convolution kernel size of the fourth and fifth layers is , the step size is 1, and the expansion ratio is 2. Therefore, the size of the original input occluded facade image is downsampled twice from becomes deep representation of .

[0039] Step 2.2: In the latent feature space, the deep representation of the occluded facade image is subjected to preliminary feature comprehensive analysis by the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

[0040] The mainstream convolution processing module includes an LSCA-M convolution processing module. The LSCA-M convolution processing module is composed of two parallel branches: a short-distance convolution processing branch and a long-distance convolution processing branch.

[0041] The short-distance convolution processing branch includes a size of The horizontal one-dimensional convolution of size is A vertical one-dimensional convolution of size Three parallel convolutions of square convolution;

[0042] The long-distance convolution processing branch consists of (expansion rate 3, size A horizontal strip convolution of size The combined convolution blocks of vertical strip convolutions in series to achieve a processing effect similar to horizontal rectangular convolution) and (with a dilation rate of 3 and a size of The vertical strip convolution and size are The proposed method consists of a combined convolution block consisting of two horizontal strip convolution blocks connected in series. The two combined convolution blocks process the feature maps in parallel to achieve a similar effect to vertical rectangular convolution. This method uses convolution processing modes of various scales and shapes to fully integrate and consider the feature differences in the image, thereby improving image classification accuracy.

[0043] Step 3: Send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features; specifically including:

[0044] The image deep representation enhanced by the mainstream convolution processing module (LSCA-M) is further represented by two auxiliary modules (as shown in the first auxiliary processing module and the second auxiliary processing module) guided by the loss function, which focus on different features and regular patterns in the deep layer. Figure 3 The two auxiliary processing modules are both composed of an average pooling layer with a step of 2, an LSCA-S module, and a linear interpolation upsampling layer. The LSCA-S module is a variant of the LSCA-M module, and since it processes on smaller feature maps, only the kernel size changes, and some redundant operations are deleted. The LSCA-S module also contains a short-distance processing branch and a long-distance processing branch. Specifically, the short-distance processing branch of the LSCA-S module contains a horizontal one-dimensional convolution with a size of a certain parameter, a vertical one-dimensional convolution with a size of a certain parameter, and a convolution; the long-distance processing branch of the LSCA-S module contains an approximate horizontal rectangular convolution with a size of a certain parameter and a dilation rate of 3, and an approximate vertical rectangular convolution with a size of a certain parameter and a dilation rate of 3 connected in parallel.

[0045] Step 4: The first multi-scale enhanced representation with distinctive hidden layer features and the second multi-scale enhanced representation are processed by the first auxiliary decoder and the second auxiliary decoder respectively, to obtain the binary classification semantic prediction map focusing on the regular component and the binary classification semantic prediction map focusing on the background; the first-stage enhanced image deep representation and the first multi-scale enhanced representation and the second multi-scale enhanced representation are superimposed and processed by the main decoder to obtain the all-class semantic prediction map.

[0046] Step 4 specifically includes:

[0047] Step S4.1: The one-time upsampling process of the two auxiliary decoders with different outputs and the same structure contains a convolution with a convolution kernel size of 3x3 and a bilinear interpolation upsampling layer with a dilation of 2, a total of 3 such upsampling processes, the first time the 16x16x512 feature map is upsampled to a 32x32x256 feature map, the second time the 32x32x256 feature map is upsampled to a 64x64x128 feature map, and the third time the 64x64x128 feature map is upsampled to a 128x128x64 feature map. Finally, the 128x128x64 feature map is output as a 128x128x2 semantic label prediction map through a linear layer and a softmax activation function. The output of the first auxiliary processing module through the binary classification prediction map output by the first auxiliary decoder pays more attention to the set regular category, and the output of the second auxiliary processing module through the second auxiliary decoder pays more attention to other categories (background). The two binary classification semantic prediction maps output by the two auxiliary decoders, which focus on different categories, will be used as the object for calculating the discrimination loss in step five.

[0048] Step 4.2: The main decoder performs an upsampling pass consisting of a 3×3 convolution followed by a bilinear interpolation upsampling layer with a dilation factor of 2. This upsampling process is performed three times: the first upsampling of the 32×32×512 feature map to a 64×64×256 feature map, the second upsampling of the 64×64×256 feature map to a 128×128×128 feature map, and the third upsampling of the 128×128×128 feature map to a 256×256×64 feature map. Finally, a linear layer and a softmax activation function are used to convert the 256×256×64 feature map into a 256×256×numclass semantic label prediction map, where numclass is the number of classes of all components in the facade image. The output of the main decoder is the final predicted labels for all classes, which are used to calculate the multi-class cross entropy loss in step 5.

[0049] Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

[0050] Step 5 may specifically include:

[0051] Step 5.1: The binary semantic prediction maps output by the two auxiliary decoders are guided by a discriminative loss function. Specifically, the first auxiliary processing module focuses on a specific category, such as windows. When calculating the binary cross-entropy loss, a greater weight is assigned to pixel regions whose true value is a window. This helps the model better identify and understand this specific category, thereby improving the model's prediction accuracy for this category. Meanwhile, the second auxiliary processing module focuses on other categories. When calculating the binary cross-entropy loss, a greater weight is assigned to pixel regions whose true value is not a window. The discriminative loss is obtained by summing the binary cross-entropy losses calculated for the outputs of the two auxiliary decoders. This helps the model better identify and understand non-window categories, thereby improving the model's prediction accuracy for these categories.

[0052] ,

[0053] Where N is the total number of input samples. is the true binary classification label value. Represents the predicted probabilities of different classes of the network.

[0054] Step 5.2: The output semantic prediction maps of all categories are guided using the multi-class cross loss function.

[0055] ,

[0056] Where N is the total number of input samples and C is the total number of categories for multi-class prediction. is the true multi-classification label value, Represents the predicted probabilities of different classes of the network.

[0057] Ultimately, the trained network can accept a facade image with or without occlusions as input. To optimize memory usage and increase prediction speed, the auxiliary decoders are used only during the training phase to guide network training. The two auxiliary decoders and their outputs are not required during the actual prediction process. The main decoder directly generates a multi-class semantic label prediction map, simplifying the prediction process.

[0058] In summary, the present invention analyzes occluded facade images through a convolution-based image segmentation network, which has important application value in the fields of image segmentation and facade analysis.

[0059] In this embodiment, each step is described in the above-mentioned order. Those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously or in a reversed order. These simple changes are within the scope of protection of the present invention.

[0060] like Figure 4 As shown in the figure, the facade analysis and prediction effects of the deep network occlusion facade perception method based on feature discrimination learning proposed in the present invention on three different datasets ECP, Ruemonge2014 and CMP. In the figure, the first row shows the input facade image, the second row presents the corresponding real multi-classification semantic label, and the third row shows the predicted label map of the method of the present invention. Observing the first three columns, these input images are facade images without occlusion. It can be found that the prediction results of the present invention are basically consistent with the real labels and have high accuracy. The middle three columns show facade images in low-light scenes. Although the ambient light is weak, the method of the present invention can still make accurate predictions by utilizing powerful long-range and short-range contextual feature representations. In the images of the last three columns, there are many categories, but even in the face of complex situations, the method of the present invention still shows excellent prediction performance.

[0061] like Figure 5As shown, based on the same technical concept as the method embodiment, an occluded facade perception system based on a feature-differentiated learning deep network is also provided. The system comprises at least: an input source acquisition module and a parsing network module. The acquisition module is configured to acquire the occluded facade image. The parsing network module is configured to map the occluded facade image to an implicit feature space. In this implicit feature space, based on the semantic information of the training image set and the surrounding unoccluded area, the module can perform deep parsing of the occluded image blocks through convolution operations. This parsing process is based on prior knowledge, that is, the regular arrangement and shape of the existing facade components, so that the image blocks can be reorganized in a structured manner. Through this parsing and reorganization operation, a final semantic prediction image can be obtained, which has high accuracy and credibility for the prediction of the occluded area. The training module includes a discrimination loss function submodule and a multi-category cross entropy loss function submodule for network training.

[0062] In an optional embodiment, the parsing network module specifically includes: an image deep representation submodule, a multi-scale enhanced representation submodule, and a semantic prediction map acquisition submodule. Among them, the image deep representation submodule is configured to downsample the damaged image and obtain a deep representation in the latent feature space. The multi-scale enhanced representation submodule is configured to perform preliminary feature shape and rule analysis in the latent feature space, and perform certain repairs on the occluded area to obtain an enhanced multi-scale facade feature representation. The semantic prediction map acquisition upsamples the different feature maps output to obtain a distinguishing semantic map with different emphases on the rule category of interest and the background, and a multi-classification semantic map with prediction labels for all categories.

[0063] In an optional embodiment, the training module specifically includes: a discriminative loss function submodule and a multi-category cross entropy loss function submodule. The discriminative loss function submodule is configured to assign different attention components to the semantic maps upsampled after processing the two auxiliary streams. Taking the regular category window as an example, the two-category semantic map output by auxiliary stream 1 assigns a greater weight to the pixel area whose true value is the window when calculating the cross entropy loss of the two categories. The two-category semantic map output by auxiliary stream 2 assigns a greater weight to the pixel area whose true value is not the window when calculating the cross entropy loss of the two categories. The multi-category cross entropy loss function submodule is configured to calculate the multi-category cross loss of the entire semantic prediction map and the true value label in a supervised learning manner using the semantic map of all component categories that are finally output.

[0064] The present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0065] The present invention provides a computer-readable storage medium having executable instructions stored thereon. When the instructions are executed by a processor, the processor can implement the aforementioned occluded facade perception method based on a feature-differentiated learning deep network.

[0066] It should be noted that the occluded facade perception system based on feature-differentiated learning deep network provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing semantic prediction on occluded images. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0067] The above-mentioned system embodiment can be used to execute the above-mentioned method embodiment. Its technical principles, technical problems solved and technical effects produced are similar. Technical personnel in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0068] It should be noted that the above describes the system embodiment and method embodiment of the present invention separately, but the details described in one embodiment can also be applied to another embodiment. The names of the modules and steps involved in the embodiments of the present invention are merely for distinguishing the modules or steps and are not to be regarded as improper limitations of the present invention. Those skilled in the art should understand that the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be combined into one module, or they can be further split into multiple sub-modules.

[0069] The above is a detailed introduction to the technical solutions provided by the embodiments of the present invention. Although this article uses specific examples to illustrate the principles and implementation methods of the present invention, the description of the above embodiments is only intended to help understand the principles of the embodiments of the present invention; at the same time, those skilled in the art will make changes within the specific implementation methods and application scope based on the embodiments of the present invention.

[0070] It should be noted that the marks and texts in the accompanying drawings are only for the purpose of more clearly illustrating the present invention and are not to be regarded as improperly limiting the scope of protection of the present invention.

[0071] The terms "comprises," "includes," or any other similar terms are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to those processes, methods, articles, or apparatus / devices. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of additional elements in the process, method, article, or apparatus / device that includes the elements, i.e., the meaning of "comprises a" also encompasses the meaning of "comprises another."

[0072] The various steps of the present invention can be implemented using a general-purpose computing device. For example, they can be centralized on a single computing device, such as a personal computer, a server computer, a handheld or portable device, a tablet device, or a multi-processor device, or they can be distributed across a network of multiple computing devices. The steps shown or described can be performed in a different order than shown, or they can be implemented as separate integrated circuit modules, or multiple modules or steps can be implemented as a single integrated circuit module. Therefore, the present invention is not limited to any specific hardware and software or any combination thereof.

[0073] The methods provided herein can be implemented using programmable logic devices (PLDs) or as computer program software or program modules (including routines, programs, objects, components, or data structures that perform specific tasks or implement specific abstract data types). For example, according to embodiments of the present invention, such a computer program product can be implemented as a computer program product. Running the computer program product causes a computer to execute the methods described herein. The computer program product includes a computer-readable storage medium containing computer program logic or code for implementing the methods. The computer-readable storage medium can be a built-in medium installed in a computer or a removable medium that can be removed from the computer (e.g., a hot-swappable storage device). Built-in media include, but are not limited to, rewritable non-volatile memory, such as RAM, ROM, flash memory, and hard disks. Removable media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or removable hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0074] The present invention is not limited to the above-mentioned embodiments. Any modification, improvement or substitution that can be conceived by those skilled in the art without departing from the essence of the present invention shall fall within the scope of the present invention.

[0075] While the forgoing detailed description has shown, described, and pointed out the fundamental novel features of the application as applied to various embodiments, it will be understood that various omissions and substitutions and changes in the form and details of the systems described can be made by those skilled in the art without departing from the spirit of the application.

Claims

1. A method for occluded facade perception based on feature-differentiated learning deep network, characterized in that: The method comprises: Step 1: Obtain a set of occluded facade images as the input source of the deep network; Step 2: Input the occluded facade image set into the deep network. The deep network predicts the occluded area based on the contextual semantic environment and passes through the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement. Step 3: The deep representation of the image after the first stage enhancement is sent to the first auxiliary processing module and the second auxiliary processing module for processing, respectively, to obtain the first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features; Step 4: The first multi-scale enhanced representation and the second multi-scale enhanced representation with discriminative hidden layer features are processed by the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the attention rule component and a binary semantic prediction map of the attention background; the deep representation of the image enhanced in the first stage is superimposed with the first multi-scale enhanced representation and the second multi-scale enhanced representation, and processed by the main decoder to obtain a semantic prediction map of all categories; Step 5: Train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

2. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 1, characterized in that: The step 1 specifically includes: The artificially synthesized occluded facade image is determined according to the following formula: ， in, represents the occluded facade image, Represents a clean facade image, represents the occlusion mask, represents pixel-by-pixel multiplication, Indicates the original image of the currently selected number of leaves.

3. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 1, characterized in that: The step 2 specifically includes: Step 2.1: The deep network uses an encoder composed of five dilated fully convolutional layers to project the input occluded facade image into the latent feature space to obtain a deep representation of the occluded facade image; Step 2.2: In the latent feature space, the deep representation of the occluded facade image is subjected to preliminary feature comprehensive analysis by the mainstream convolution processing module to obtain the deep representation of the image after the first stage enhancement.

4. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 3 is characterized in that: The mainstream convolution processing module includes an LSCA-M submodule that performs comprehensive feature understanding of the deep representation of the image downsampled by the encoder at short and long distances.

5. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 1 is characterized in that: The first auxiliary processing module and the second auxiliary processing module have the same structure, both including a pooling layer submodule for reducing the size of the feature map, an LSCA-S submodule, and a bilinear upsampling layer submodule for increasing the size of the feature map.

6. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 1, characterized in that: The main decoder includes a bilinear upsampling layer submodule that increases the size of the feature map.

7. The method for occluded facade perception based on feature-differentiated learning deep network according to claim 1 is characterized in that: The step 5 specifically includes: Step 5.1: The binary semantic prediction maps output by the two auxiliary decoders are guided using the discriminative loss function; Step 5.2: All category semantic prediction maps are guided using the multi-class cross loss function.

8. An occluded facade perception system based on feature-differentiated learning deep network, characterized in that: The system comprises: The input source acquisition module is used to obtain a set of occluded facade images as the input source of the deep network; The deep image representation module is used to input the set of occluded facade images into a deep network. The deep network predicts the occluded area based on the contextual semantic environment and obtains the first-stage enhanced deep image representation after the mainstream convolution processing module. A multi-scale enhanced representation module is used to send the deep representation of the image after the first stage enhancement to the first auxiliary processing module and the second auxiliary processing module for processing, thereby obtaining a first multi-scale enhanced representation and a second multi-scale enhanced representation with discriminative hidden layer features; The semantic prediction map acquisition module is used to process the first multi-scale enhanced representation and the second multi-scale enhanced representation through the first auxiliary decoder and the second auxiliary decoder respectively to obtain a binary semantic prediction map of the focus rule component and a binary semantic prediction map of the focus background; the deep representation of the image enhanced in the first stage is superimposed with the first multi-scale enhanced representation and the second multi-scale enhanced representation, and processed through the main decoder to obtain semantic prediction maps of all categories; The training module is used to train the deep network based on the binary semantic prediction map and the semantic prediction map of all categories and save the trained deep network.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; Among them, when one or more programs are executed by the one or more processors, the one or more processors implement the occluded facade perception method based on feature-differentiated learning deep network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement a method for perceiving an obstructed facade based on a feature-differentiated learning deep network as described in any one of claims 1 to 7.

Images