A glass detection method based on deep learning and ghost phenomenon

Abstract

The application relates to a glass detection method based on deep learning and ghosting, and relates to the technical field of computer vision. The method comprises the following steps: performing glass detection based on a single original input image, the image being a single RGB image; extracting ghost features through a deep learning method of a backbone network based on the original input image to obtain a ghost area prediction map; connecting the ghost area prediction map with the original input image channel, extracting glass features based on the backbone network under the guidance of ghost clues, then performing glass feature decoding and glass area segmentation based on a convolutional neural network; and outputting a glass area prediction map. Compared with the prior art, the application performs glass detection based on a single image, and is more widely applicable. The backbone network can more accurately and efficiently extract ghost features and glass area features, and the ghosting phenomenon can be used to more accurately locate the glass area, obtain a high-quality glass area prediction map, and has good robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a glass detection method based on deep learning and ghosting phenomenon. BACKGROUND

[0002] Glass detection has recently attracted extensive attention. Glass surfaces, including glass windows, glass doors and glass walls, are ubiquitous in our daily life in indoor and outdoor scenes. However, since they are transparent surfaces, there is usually no specific visual image, and the information presented depends largely on the scene behind them. Due to the lack of consistent visual appearance and special functions of glass, computer vision-based systems such as robots and drones can easily overlook glass surfaces, and often the detected glass region is not the glass surface but the scene behind the glass, which will affect their normal work. Therefore, accurately detecting glass surfaces is crucial for many computer vision-based systems.

[0003] At the same time, with the development of computer technology and the wide application of computer vision principles, deep learning has rapidly developed in the field of computer vision based on its powerful learning ability and feature expression ability, and has quickly replaced the previous artificial construction of features based on prior knowledge. In recent years, deep learning methods based on transformers have achieved results exceeding convolutional neural networks in many fields.

[0004] Existing deep learning-based methods use contextual contrast information, but do not mine useful clues for the physical properties of glass. Glass detection using reflection also has great limitations, because reflection is not a physical property unique to glass regions, and the reflection produced by smooth surfaces such as walls, floors and displays will affect the accuracy of the detection results of glass regions. SUMMARY

[0005] The technical problem to be solved by the present application is that the detection results of the prior art for detecting glass regions have low accuracy. The purpose of the present application is to provide a glass detection method based on deep learning and ghosting phenomenon, which uses a deep learning method to more effectively extract global features and a glass detection method based on ghosting phenomenon to more accurately detect glass regions to obtain high-quality detection results, and has good robustness.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] On the one hand, a glass detection method based on deep learning and ghosting phenomenon includes the following steps:

[0008] Glass detection based on a single original input image, the image being a single RGB image;

[0009] extracting ghost features based on a deep learning method of a backbone network from an original input image, and calculating a ghost region prediction map;

[0010] After the ghost region prediction map is connected with the original input image channel, glass features are extracted based on the backbone network under the guidance of ghost clues, and then glass feature decoding is performed based on a convolutional neural network to obtain a glass region segmentation result;

[0011] Based on the glass region segmentation result, a glass region prediction map is output.

[0012] The ghost region prediction map includes the following steps:

[0013] Based on the backbone network, multi-scale features are obtained;

[0014] Based on the obtained multi-scale features, a double reflection estimation module is input to obtain an offset estimation map to detect a ghost region, wherein the double reflection estimation module obtains a first reflection feature and a second reflection feature through a first reflection detection and a second reflection detection;

[0015] The first reflection feature, the second reflection feature, and the offset estimation map are fused to obtain ghost features, and a high-quality ghost region prediction map is obtained through a decoder based on a convolutional neural network.

[0016] The double reflection estimation module process includes the following steps:

[0017] Based on the multi-scale features obtained by the backbone network, a double reflection estimation module is input;

[0018] Based on the multi-scale features, a first reflection feature and a first reflection region prediction map are obtained through a first detection, and a second reflection feature and a second reflection region prediction map are obtained through a second detection;

[0019] The first reflection feature and the second reflection feature are constrained through deformable convolution;

[0020] The first reflection region prediction map and the second reflection prediction map are input to an encoder-decoder structure, and an offset estimation map is obtained through an encoder;

[0021] The ghost features obtained by fusing the first reflection feature, the second reflection feature, and the offset estimation map are input to a decoder to obtain a ghost region prediction map.

[0022] The feature constraint subtracts the first reflection feature from the second reflection feature through deformable convolution, and the loss function used is calculated according to the following formula:

[0023]

[0024] wherein for a deformable convolution operation, for a first reflection feature under a corresponding i scale, for a second reflection feature under a corresponding i scale.

[0025] The backbone network is a Swin-Transformer.

[0026] The present application extracts global features based on the Swin-Transformer, and comprehensively considers the physical characteristics of the ghost itself, designs a double reflection estimation module to predict the offset map, which can improve the accuracy of the detected ghost area. The feature constraint performed is also conducive to the extraction of ghost features.

[0027] In another aspect, a glass detection system based on deep learning and ghost phenomenon, the system is suitable for a glass detection method based on deep learning and ghost phenomenon, the system comprises:

[0028] An acquisition module is configured to acquire a single original input image.

[0029] A ghost detection module is configured to extract ghost features based on the original input image through a deep learning method of a backbone network, and to calculate and acquire a ghost area prediction map.

[0030] A glass segmentation module is configured to connect the ghost area prediction map with the original input image channel, extract glass features under the guidance of ghost clues based on the backbone network, and then perform glass feature decoding and glass area segmentation based on a convolutional neural network.

[0031] An output module is configured to output a glass area prediction map.

[0032] The glass segmentation module is in a U-shaped structure, and comprises an encoding part and a decoding part.

[0033] The glass segmentation module of the present application is designed in a U-shaped structure, which is different from the traditional U-Net and most methods based on convolutional neural networks. The present application uses a combination of Swin-Transformer and convolution, and the global features extracted by the transformer are conducive to the positioning of the potential glass area. The convolutional neural network is used for feature fusion and progressive decoding, and finally a high-quality glass area prediction map is obtained.

[0034] The technical scheme provided by the present application has at least the following beneficial effects:

[0035] Compared with the prior art, the glass detection based on a single image is more widely applicable. In network construction, the backbone network is used to construct the network, which can more accurately and efficiently extract ghost features and glass region features. Meanwhile, by using the special visual clue of ghost, the glass region can be more accurately located, and a high-quality glass region prediction map can be obtained, which has good robustness. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0037] Figure 1 A glass detection method based on deep learning and ghost phenomenon is shown in the flowchart;

[0038] Figure 2 A flowchart of the ghost detection module provided by an exemplary embodiment of the present application is shown;

[0039] Figure 3 A process diagram of the double reflection estimation module provided by an exemplary embodiment of the present application is shown;

[0040] Figure 4 A flowchart of the double reflection estimation module provided by an exemplary embodiment of the present application is shown;

[0041] Figure 5 A structural block diagram of a glass detection system based on deep learning and ghost phenomenon provided by an exemplary embodiment of the present application is shown;

[0042] Figure 6 A structural block diagram of a double reflection estimation module in a glass detection system based on deep learning and ghost phenomenon provided by an exemplary embodiment of the present application is shown;

[0043] Figure 7 A connection diagram of the glass detection network provided by an exemplary embodiment of the present application is shown;

[0044] Figure 8 The experimental result graph is shown, in which the first column Input is the input real scene picture with glass region; the second column OurGhosts is the 2D mask of the ghost region obtained by an exemplary embodiment of the present application; the third column Ours is the 2D mask of the glass region predicted and obtained by an exemplary embodiment of the present application, and the fourth column GT is the true value graph of the glass region mask. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0046] The present application will be further described below in combination with the drawings and embodiments.

[0047] First, the terms involved in the embodiments of the present application are briefly introduced:

[0048] Ghosting effect is an inherent property of glass surface and always appears on the glass surface. This is due to the glass pane glass having two contact surfaces (on both sides), resulting in two attenuated slightly offset reflections, and using ghosting information can effectively guide the detection of the glass region. Since ghosting only appears in the glass region in daily life, it can effectively provide guidance and support for detecting glass.

[0049] SwinTransformer is a deep learning method based on Transformer.

[0050] Figure 1 An example embodiment of the present application provides a glass detection method based on deep learning and ghosting phenomenon. The method comprises the following steps:

[0051] Step 101, glass detection based on a single original input image, the image being a single RGB image.

[0052] Step 102, extracting ghosting features based on the original input image through the deep learning method of the backbone network to calculate and obtain the ghosting region prediction map.

[0053] Step 103, after connecting the ghosting region prediction map with the original input image channel, extracting glass features based on the backbone network under the guidance of ghosting clues, and then decoding glass features and glass region segmentation results based on the convolutional neural network.

[0054] Step 104, based on the glass region segmentation result, outputting a glass region prediction map. Repeat the training until a high-quality glass region prediction map is obtained.

[0055] The specific process is as follows:

[0056] The glass detection is performed based on a single original input image, and then a ghost detection module is used to extract ghost features by using a backbone network, i.e., a Swin-Transformer, to obtain the features input into a double reflection estimation module, two branches are used to detect primary reflection and secondary reflection; a deformable convolution block is used to align the features of the primary reflection with the reflected features to accurately estimate the primary reflection and the secondary reflection (double reflection estimation block). The predicted map of the estimation of the primary reflection and the secondary reflection is up-sampled to the original resolution and input into an encoder-decoder structure to estimate a displacement map, i.e., an offset estimation map. The obtained displacement map is down-sampled and connected with the feature map of the primary reflection and the secondary reflection, and after fusion, the connected map is input into a decoder to decode a ghost region prediction map. Then, a glass segmentation module is used to segment the glass region under the guidance of the ghost region. After the ghost region is connected with the input image channel, the connected map is input into another backbone network to extract glass features, and then a convolutional neural network is used for feature decoding and obtaining a glass region segmentation structure, and the training is repeated until a high-quality glass region prediction map is obtained.

[0057] In summary, the glass detection method based on deep learning and ghost phenomenon provided in the application not only improves the method compared with the prior art, but also analyzes the physical properties of glass, which is more unique. Since ghosts only appear in the glass region in daily life, the method can effectively guide and support the detection of glass.

[0058] The global features are extracted by the method based on SwinTransformer, and the physical properties of the ghost itself are comprehensively considered, a double reflection estimation module is designed to predict an offset map, which can improve the accuracy of the detected ghost region. The feature constraint is also beneficial to the extraction of ghost features.

[0059] The structure of the glass segmentation module is designed as a U-shaped structure, i.e., an encoder-decoder structure, which is different from the traditional U-Net and most methods based on convolutional neural networks. The Swin-Transformer and convolution are combined, the global features extracted by the transformer are beneficial to the positioning of the potential glass region, the convolutional neural network is used for feature fusion and progressive decoding, and finally a high-quality glass region prediction map is obtained.

[0060] The application detects glass based on a single image, and is more widely applicable. Based on a single image, it is different from the multi-modal input method of processing multiple images at the same time, which needs to use additional input devices such as infrared cameras, polarization cameras, and depth cameras. Compared with using a single camera, the use scene has greater limitations and poorer applicability.

[0061] Figure 2A flowchart of a ghost detection module provided by an example embodiment of the present application is shown. The method includes the following steps:

[0062] In step 201, multi-scale features are obtained through a backbone network.

[0063] In step 202, a displacement map is estimated based on the obtained multi-scale features to detect a ghost region, wherein the double reflection estimation module uses two branches to detect primary reflection and secondary reflection to obtain primary reflection features and secondary reflection features.

[0064] In step 203, the primary reflection features, the secondary reflection features, and the displacement map are fused to obtain ghost features, and a high-quality ghost region prediction map is obtained through a decoder based on a convolutional neural network.

[0065] Figure 3 And Figure 4 A double reflection estimation module provided by an example embodiment of the present application is shown. The process includes the following steps:

[0066] In step 301, the double reflection estimation module is input based on the multi-scale features obtained by the backbone network.

[0067] In step 302, based on the multi-scale features, the primary reflection features and the primary reflection region prediction map are obtained through primary detection, and the secondary reflection features and the secondary reflection region prediction map are obtained through secondary detection.

[0068] In step 303, the primary reflection features and the secondary reflection features are aligned by a deformable convolution block to accurately estimate the primary reflection and the secondary reflection.

[0069] In step 304, the primary reflection region prediction map and the secondary reflection prediction map are sampled to the original resolution and input to an encoder-decoder structure. An offset estimation map is obtained through an encoder, and the offset estimation map is an estimated displacement map.

[0070] In step 305, the ghost features obtained by fusing the primary reflection features, the secondary reflection features, and the offset estimation map are input to a decoder, and the ghost region prediction map is decoded by the decoder.

[0071] Figure 5 A structural block diagram of a glass detection system based on deep learning and ghost phenomenon provided by an example embodiment of the present application is shown. The system includes an acquisition module 410, a ghost detection module 420, a glass segmentation module 430, and an output module 440.

[0072] The acquisition module 410 is configured to acquire a single original input image.

[0073] The ghost detection module 420 is configured to extract ghost features based on a deep learning method of the backbone network based on the original input image, and calculate a ghost region prediction map.

[0074] The glass segmentation module 430 is configured to connect the ghost region prediction map with the original input image channel, extract glass features based on the backbone network under the guidance of ghost clues, and then perform glass feature decoding based on a convolutional neural network and glass region segmentation.

[0075] The output module 440 is configured to output a glass region prediction map.

[0076] Figure 6 A structural block diagram of a double reflection estimation module in a glass detection system based on deep learning and ghosting phenomenon is shown.

[0077] The ghost detection module can first obtain multi-scale features by using the backbone network, then input the multi-scale features into ghost features in the image through a specially designed double reflection estimation module, and finally obtain a ghost region prediction map through the decoder.

[0078] The ghost detection module 420 includes a backbone network 510, a double reflection estimation module 520, and a decoder 530. Preferably, the backbone network 510 is a SwinTransformer network.

[0079] The backbone network 510 is configured to detect potential glass regions. Preferably, the backbone network uses an existing SwinTransformer. First, since the ghost effect is usually observed as a repetition of the edge in the input image, the application takes advantage of the advantages of SwinTransformer in extracting low-level features and learning long-distance dependencies. Second, SwinTransformer can model the region correlation in a local-to-global hierarchical manner, which helps to handle the appearance changes (e.g., intensity and shape) of the ghost effect.

[0080] Compared with existing methods, the transformer has an advantage in extracting long-distance dependent features. Compared with traditional CNN-based methods, the transformer can obtain a larger receptive field at a relatively shallow layer of the network, and can have more advantages in extracting features conducive to glass detection.

[0081] The double-reflection estimation module 520, on the premise that the multi-scale features obtained from the backbone network have been obtained, detects the ghosting effect by using the double-reflection estimation module to obtain the offset estimation map at the multi-scale. The double-reflection estimation is used to detect the existence of the reflection displacement. The present application first detects any two main reflections, and then estimates the offset between the reflection layers. Such a design brings two actual advantages. First, based on the ghosting phenomenon, the present application proposes a model that can handle any type of glass surface without considering the number of glass regions in the input image. Second, the designed module does not need to accurately estimate the offset, so the present application does not provide the ground truth of the real scene, and is only supervised on the synthetic scene.

[0082] The present application uses two branches to detect the primary reflection and the secondary reflection. The present application uses a deformable convolution block to align the feature displacement of the primary reflection with the feature of the secondary reflection to accurately estimate the primary reflection and the secondary reflection. The predicted map of the estimation of the primary reflection and the secondary reflection is first up-sampled to the original resolution, and then input to the encoder-decoder structure to estimate the displacement map, which is the offset estimation map. The non-0 value in the displacement map can indicate the existence of ghosting effect in the region. The obtained displacement map can also provide a strong clue to distinguish the ghosting effect from the single reflection. Finally, the estimated displacement map is down-sampled and connected with the feature map of the primary reflection and the secondary reflection, and after fusion, it is input to the decoder.

[0083] The decoder part uses a convolutional neural network to input the features output by the double-reflection estimation module into the decoder to obtain the final ghosting region prediction result.

[0084] The convolutional network is used for decoding and fusing the multi-scale features extracted by the backbone network, and the fusion of the multi-scale features is beneficial to detecting glass regions of different sizes.

[0085] The loss function used by the supervised part involved in the network of the present application is as follows.

[0086] The loss function used for the prediction supervision of the primary reflection 2D mask, the secondary reflection 2D mask and the ghosting region 2D mask is the BCE loss function, as follows:

[0087] Formula one:

[0088] Where i indicates the scale index of the predicted prediction mask, s represents the total number of scales. M represents the predicted 2D mask, represents the ground truth.

[0089] The loss function used by the feature constraint part is the mean square error loss function, and the specific calculation formula is as follows:

[0090] Formula two:

[0091] wherein is a deformable convolution operation, is a feature of a first reflection under a corresponding i scale, is a feature of a second reflection under a corresponding scale i.

[0092] Figure 7 A glass detection network connection diagram provided by an example embodiment of the application is shown. The glass segmentation module is a U-shaped network structure, and the module structure is an encoder-decoder structure, which is used to better detect glasses of different sizes.

[0093] The encoder uses SwinTransformer to obtain four layers of feature representations of different scales. The input is first positionally encoded, and the number of stacked SwinTransformer blocks in the network from shallow to deep is 2, 2, 16, and 2, respectively. The decoder combines the feature maps obtained in the encoding stage with the feature maps obtained in the decoding stage through channel linking, combines deep and shallow features, refines the image, and predicts the glass region according to the obtained feature map.

[0094] Loss function:

[0095] Equation three:

[0096] The loss function of the glass segmentation module includes three parts, namely the BCE loss function, the SSIM loss function, and the IOU loss. The final loss function is the sum of the three.

[0097] Figure 8 Experimental results are shown in the following figures. The application is described and illustrated by the following specific experiments:

[0098] The spatial resolution of the input image is 384x384. The training is performed on a server equipped with Intel i9-10900X 10 core / 3.7G / 19.25M CPU, 16G memory, and NVIDIA RTX3090 24GB graphics memory GPU. The network training environment is python3.6.13 and pytorch1.7.1, the iteration number is set to 200, and the batch size is set to 2. The Adam optimizer is used to train the network. The learning rate of the ghost detection module is set to 0.00001, and the learning rate of the glass segmentation module is set to 0.00001.

[0099] The network training process of the present application is as follows: first, the ghost feature is extracted by using the backbone network. The obtained feature is input into the double reflection estimation module, and two branches are used to detect the primary reflection and the secondary reflection. Then, the primary reflection feature is aligned with the secondary reflection feature using deformable convolution to accurately estimate the primary reflection and the secondary reflection. First, the estimated prediction map of the primary reflection and the secondary reflection is up-sampled to the original resolution, and then input into the encoder-decoder structure to obtain the offset estimation map. Finally, the obtained offset estimation map is down-sampled and connected with the feature map of the primary reflection and the secondary reflection, and after fusion, it is input into the decoder to decode the ghost region.

[0100] The glass segmentation module segments the glass region under the guidance of the ghost region. Specifically, after connecting the obtained ghost region with the input image channel, the glass feature is extracted by inputting it into another backbone network, and then the feature is decoded by using the convolutional neural network and the glass region segmentation result is obtained. Finally, such repeated training is performed until a high-quality glass region prediction map is obtained.

[0101] Figure 8 The results of the experiment are shown in the figure, which shows the qualitative evaluation of the present method in real scenes, from Figure 8 As can be seen from the above, the present application can not only accurately detect the ghost region in real scenes, but also accurately predict the glass region. The first and second rows show scenes of large glass under different lighting conditions, and the present method can accurately capture the ghost phenomenon and detect the glass region. The third and fourth scenes show complex outdoor scenes with multiple glass regions, and the third row of the scene has a small glass region at the edge due to scene limitations, and the fourth row has a small glass region due to occlusion. The fifth row shows the results of glass detection in outdoor scenes, and the present method can also accurately detect glass. The present application can still accurately predict these regions under the condition that most of the glass has been detected. It is proved that the present application has good practicability and universality.

[0102] In summary, the glass detection method based on deep learning and ghost phenomenon provided by the embodiment of the present application has the advantages that, compared with the prior art, the present application detects glass based on a single image, and is more widely applicable. In terms of network structure, a powerful Transformer is used to construct the network, which can more accurately and efficiently extract ghost features and glass region features. At the same time, by using the special visual clue of ghost, the glass region can be more accurately located, and a high-quality glass region prediction map can be obtained, which has good robustness.

[0103] The above merely is the optional embodiment of the present application, and does not use to limit the present application, any modification, equivalent replacement, improvement and so on, which are made in the spirit and principle of the present application, should be included in the protection scope of the present application.

Claims

1. A glass detection method based on deep learning and ghosting phenomenon, comprising the following steps: Glass detection is performed based on a single original input image, where the image is a single RGB image; Based on the original input image, a deep learning method using a backbone network is used to extract ghosting features and calculate and obtain a predicted image of the ghosting region. After connecting the predicted ghosting region map with the original input image channel, glass features are extracted based on the backbone network under the guidance of ghosting cues. Then, glass feature decoding and glass region segmentation results are obtained based on the convolutional neural network. Based on the segmentation results of the glass region, output a predicted map of the glass region; The process of obtaining the predicted image of the ghosting region includes the following steps: Multi-scale features are obtained based on backbone networks; The dual reflection estimation module obtains an offset estimation map based on the acquired multi-scale feature input to detect ghosting regions. The dual reflection estimation module obtains primary reflection features and secondary reflection features through primary reflection detection and secondary reflection detection. By fusing primary reflection features, secondary reflection features, and offset estimation maps, ghosting features are obtained. A high-quality ghosting region prediction map is then obtained through a convolutional neural network-based decoder.

2. The glass detection method based on deep learning and ghosting phenomenon according to claim 1, characterized in that, The execution process of the dual reflection estimation module includes the following steps: A dual-reflection estimation module based on multi-scale feature inputs obtained from a backbone network; Based on multi-scale features, a single detection is used to obtain primary reflection features and a primary reflection region prediction map, and a second detection is used to obtain secondary reflection features and a secondary reflection region prediction map. Feature constraints are applied to primary and secondary reflection features using deformable convolution. The prediction maps of the primary reflection region and the secondary reflection region are input into the encoder-decoder structure, and the offset estimation map is obtained through the encoder. The ghosting features obtained by fusing primary reflection features, secondary reflection features, and offset estimation map are input into the decoder to obtain the ghosting region prediction map.

3. The glass detection method based on deep learning and ghosting phenomenon according to claim 2, characterized in that, The feature constraint is achieved by subtracting the first-order reflection feature from the second-order reflection feature through variable convolution, using the following formula for calculating the loss function: in For variable convolution operations, This corresponds to the reflection feature at scale i. This represents the secondary reflection characteristics at scale i.

4. The glass inspection method based on deep learning and ghosting phenomenon according to any one of claims 1 to 3, characterized in that, The backbone network is Swing Transformer.

5. A glass inspection system based on deep learning and ghosting phenomenon, applicable to the inspection method as described in claim 1, characterized in that, The system includes: The acquisition module is used to acquire a single raw input image; The ghosting detection module is used to extract ghosting features based on the original input image using a deep learning method with a backbone network, and to calculate and obtain a predicted image of the ghosting region. The glass segmentation module is used to connect the predicted image of the ghosting region with the original input image channel, extract glass features based on the backbone network under the guidance of ghosting cues, and then decode the glass features and segment the glass region based on the convolutional neural network. The output module is used to output a predicted map of the glass area.

6. The glass inspection system based on deep learning and ghosting phenomenon according to claim 5, characterized in that, The glass segmentation module has a U-shaped structure and includes an encoding section and a decoding section.

Images