System and method for eliminating reflected images from automobile glass

Abstract

The present invention discloses a system and method for eliminating automobile glass reflection images, comprising an image preprocessing network, a transmission image generation network, and a generated image discrimination network. The image preprocessing network performs feature extraction on an input image to obtain hypercolumn feature image data; the transmission image generation network performs deep-level processing on the hypercolumn feature image to generate a de-reflected predicted image; the generated image discrimination network feeds the predicted image and a real non-reflection image sample corresponding to the input image into a multivariate loss function for discrimination, and eliminates the automobile glass reflection image system through continuous iterative training. The present invention achieves an effect in which the predicted image is close to the real non-reflection image, thereby achieving the purpose of removing reflection.

Description

Technical Field

[0001] The invention belongs to the field of intelligent road traffic monitoring and is a system for eliminating reflected images in automobile glass. Background Art

[0002] With the increasing application of surveillance equipment in road traffic monitoring, a large number of traffic monitoring cameras have been installed at key intersections and accident-prone areas, and an increasing number of vehicles recording traffic violations have been deployed in daily patrols. At the same time, intelligent traffic monitoring systems are placing increasingly high demands on the quality of road monitoring images. The captured images must clearly identify information such as vehicle license plates and human features. However, road monitoring equipment is often affected by reflections from vehicle windshields during imaging. This can cause the captured image to be obscured by the reflected scene inside the vehicle, weakening or even eliminating the target features to be identified. This severely impacts the judgment of occupant behavior and complicates road traffic monitoring. Therefore, eliminating the interference from glass reflections has become a pressing issue in road traffic monitoring systems.

[0003] A common method for reducing the impact of reflections on image quality is to adjust the shooting method, such as choosing a large aperture lens, placing a polarizing filter in front of the lens, changing the shooting position and angle, reducing the intensity of reflected scene light, and increasing the brightness of background light. However, these methods are subject to significant subjective influences and have limited application scenarios, making them difficult to use in practice. From the perspective of image processing algorithms for de-reflection, in an image with reflections, the reflection layer information and the background layer information are fused together, which is mathematically described as an ill-posed problem. Traditional image reflection removal methods require the establishment of complex degradation models and have poor universality for different reflection scenarios. Currently, there is no specific solution to the problem of vehicle windshield reflection interference in road monitoring scenarios. However, the rise of deep learning in the field of computer vision has provided new solutions to this problem.

[0004] The image reflection removal method based on deep learning needs to solve two key problems. The first is that the network must be able to accurately extract the position information of the reflective and non-reflective areas in the image, strengthen the non-reflective areas, and suppress the reflective areas; the second is that the network must be able to extract the characteristic channel information of the image and learn the relationship between the channels to synthesize more realistic reflection layer and transmission layer images. Existing deep learning networks lack targeted research in terms of position information and depth. Although good results have been achieved in de-reflecting flat glass, the effect of removing reflections from free-form surface automotive glass is not good. Therefore, the present invention focuses on conducting targeted research on position information and depth enhancement in the de-reflection network, and proposes a method for removing reflections from automotive glass images based on the CBAM attention residual structure, which has important theoretical significance for improving the quality of road monitoring imaging. Since this method does not require changing the status of existing monitoring hardware, it also provides a new solution for clear imaging inside road monitoring vehicles. Summary of the Invention

[0005] To address the problem of glass reflections interfering with in-vehicle imaging quality during road traffic monitoring photography, the present invention proposes a system and method for eliminating reflected images in glass. Aiming to specifically improve pixel channel position information and enhance the differences between the reflective and transmissive layers, the system improves the basic network consisting of hypercolumn feature extraction, generative networks, and multivariate loss functions, and introduces a CBAM attention machine to reduce the influence of curved glass on transmissive imaging.

[0006] The main innovations are: using CBAM ResBlock in the generative network to replace the last five dilated convolutional layers in the original network, retaining the expansion rate of the original dilated convolutional layer to avoid network degradation; the CBAM sub-module adopts a "channel attention + spatial attention" sub-module connection method, which significantly improves the effect of eliminating car glass reflections; inserting the CBAM module into the residual structure comprehensively enhances the network performance in terms of position information and depth.

[0007] The present invention is implemented by adopting the following technical solutions:

[0008] A system for eliminating reflected images from automobile glass, comprising:

[0009] Image preprocessing network, transmission image generation network and generated image discrimination network, where:

[0010] The image preprocessing network extracts features from the input image to obtain hypercolumn feature image data;

[0011] The transmission image generation network performs deep processing on the hypercolumn feature image to generate a de-reflected prediction image; wherein: the transmission image generation network includes a first convolution module, a first attention module, a second attention residual module and a second convolution module;

[0012] The first attention module is a convolutional network embedded with 5 layers of residual units; the expansion rate of the first layer of the convolutional network is 8, the expansion rate of the second layer is 16, the expansion rate of the third layer is 32, the expansion rate of the fourth layer is 64, and the expansion rate of the fifth layer is 128;

[0013] The second attention module is a convolutional network embedded with 8 layers of residual units; the expansion rate of each layer of the convolutional network is 1;

[0014] The residual unit includes a weight calculation part and an attention operation part; the weight calculation part is composed of a convolution layer and an activation function; the attention operation part is connected in series using channel attention and spatial attention;

[0015] The generated image discrimination network sends the predicted image and the real non-reflection image samples corresponding to the input image into the multivariate loss function for discrimination, and eliminates the automobile glass reflection image system through continuous iterative training.

[0016] Furthermore, the image preprocessing network adopts the VGG-19 convolutional neural network, and the convolutional neural network structure is conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 feature layers.

[0017] Furthermore, the generated image discriminant network calculates the deviation between the generated network predicted image and the real non-reflection image sample through the following multivariate loss function:

[0018] L(θ)=ω1L feat (θ)+ω2L adv (θ)+ω3L excl (θ)

[0019] Where: L(θ) is the total loss function, L feat (θ) is the feature loss function, L adv (θ) is the adversarial loss function, L excl (θ) is the exclusion loss function, θ represents the network weight, ω1, ω2, and ω3 are weighting parameters, with values ​​of ω1 = 0.1, ω2 = 0.01, and ω3 = 1.0, respectively; the three loss functions are linearly combined as the total network loss function.

[0020] The present invention is also implemented by the following technical solutions:

[0021] A method for eliminating reflection images from automobile glass, the method comprising the following steps:

[0022] Perform feature extraction on the input image to obtain hypercolumn feature image data;

[0023] A fully convolutional network with an attention residual model is used to perform deep processing on the hypercolumn feature image to generate a de-reflected predicted image; wherein: the attention residual model is composed of a first attention module and a second attention residual module;

[0024] The first attention module is a convolutional network embedded with 5 layers of residual units; the expansion rate of the first layer of the convolutional network is 8, the expansion rate of the second layer is 16, the expansion rate of the third layer is 32, the expansion rate of the fourth layer is 64, and the expansion rate of the fifth layer is 128;

[0025] The second attention module is a convolutional network embedded with 8 layers of residual units; the expansion rate of each layer of the convolutional network is 1;

[0026] The residual unit includes a weight calculation part and an attention operation part; the weight calculation part is composed of a convolution layer and an activation function; the attention operation part is connected in series using channel attention and spatial attention;

[0027] The predicted image and the real non-reflection image sample corresponding to the input image are fed into the multivariate loss function for discrimination, and the generative network model is converged through continuous iteration, where:

[0028] The deviation between the network prediction image and the real non-reflection image sample is calculated by the following multivariate loss function:

[0029] L(θ)=ω1L feat (θ)+ω2L adv (θ)+ω3L excl (θ)

[0030] Where: L(θ) is the total loss function, L feat (θ) is the feature loss function, L adv (θ) is the adversarial loss function, L excl (θ) is the exclusion loss function, θ represents the network weight, ω1, ω2, and ω3 are weighting parameters, with values ​​of ω1 = 0.1, ω2 = 0.01, and ω3 = 1.0, respectively; the three loss functions are linearly combined as the total network loss function.

[0031] Beneficial effects

[0032] 1. This paper introduces the attention residual structure and uses CBAM ResBlock to replace the last five dilated convolutional layers in the original network. During the replacement process, the expansion rate of the original dilated convolutional layer is retained. This can retain the larger receptive field of the original network to a certain extent and avoid network degradation.

[0033] 2. The present invention significantly improves the reflection elimination effect by connecting the "channel attention + spatial attention" sub-modules in series in the attention residual structure. This link adopts the "channel attention + spatial attention" method to achieve the best de-reflection processing effect. In addition, in order to extract deeper feature information in the reflection image processing layer and the transmission image processing layer, the depth of the network needs to be deepened. However, this will cause the gradient to disappear and even degradation problems. In order to avoid this situation, the present invention uses a residual structure in the end-to-end generative network, and inserts the above-mentioned "channel attention + spatial attention" CBAM module into the residual structure to learn the channel and pixel position information of the image features, and perform comprehensive enhancement in terms of position information and depth, which is recorded as CBAM ResBlock.

[0034] 3. The present invention still uses a linear combination of multivariate loss functions to guide the separation of image layers, and can simultaneously encode shallow and deep image information.

[0035] 4. The present invention achieves the effect of predicting images close to the real non-reflection image, achieving the purpose of removing reflections. In particular, to address the problem of irregular reflections on the free-form surface of automobile glass, an improved attention residual structure is applied to the generative network. On the one hand, the channel attention and spatial attention are serially used to improve the extraction effect of reflection features. On the other hand, it is added to the residual structure and deeply expanded to obtain deeper image feature information, solving the problem that the existing network depth is relatively shallow and is good at removing reflections from flat glass but not so good at removing reflections from automobile glass. The present invention can remove reflection artifact images from automobile glass and has good application prospects in the field of road traffic monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a schematic structural diagram of a system and method for eliminating reflected images in glass.

[0037] Figure 2 Comparison of the effects before and after car glass reflection treatment. DETAILED DESCRIPTION

[0038] The following combination Figures 1 and 2 The present invention is described in detail

[0039] A method for eliminating automobile glass reflection images comprises an image preprocessing network, a transmission image generation network and a generated image discrimination network.

[0040] The image preprocessing network extracts features from the input image to obtain hypercolumn feature image data. Specifically, the input image is first subjected to feature layer extraction. The 19-layer convolutional network in the VGG-19 convolutional neural network is used to convolve the input image. The conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2 feature layers are extracted and concatenated with the input image to form a hypercolumn feature structure, which serves as the input to the transmission image generation network.

[0041] The transmission image generation network is an end-to-end grid that uses a fully convolutional network with an attention residual structure to perform deep processing on the hypercolumn feature image to generate a de-reflected predicted image.

[0042] The generated network in the base network has a relatively shallow 10 layers. Further increases in the number of layers would lead to network degradation, preventing the extraction of deeper information and affecting the dereflection effect. The transmission image generation network described in this paper uses an attention residual structure instead of the original dilated convolutional layer, bringing the network depth to 18 layers. This achieves optimal dereflection effect on automotive glass.

[0043] The 18-layer network design replaces the last five dilated convolutional layers with five attention residual structures, maintaining a larger receptive field. The dilation rate of each convolution layer remains unchanged from 8, 16, 32, 64, and 128, respectively, to maintain a larger receptive field. To capture deeper image features, eight more attention residual layers are added. Since the receptive field is already guaranteed, the dilation rate of the last eight layers is set to 1, thus addressing the network degradation issue of extracting deeper information.

[0044] The attention residual structure consists of two parts. One part is the weight module, which consists of a convolutional layer, an activation function and a convolutional layer, and the expansion rate of the first convolutional layer is consistent with the expansion rate of the original position convolutional layer; the other part is inserted with an attention mechanism module, which adopts a serial method of channel attention and spatial attention. This method can achieve the best reflection and transmittance enhancement effect.

[0045] The attention residual structure is inserted into the generation grid in a hierarchical series manner, wherein the generation network has 18 layers, but is not limited to 18 layers. The number of layers can be adapted and determined according to the imaging characteristics of the road monitoring hardware equipment.

[0046] The generative image discrimination network sends the predicted image and the real non-reflection image samples corresponding to the input image into the multivariate loss function for discrimination, and through continuous iteration, the generative network model converges to achieve the effect of the predicted image close to the real non-reflection image, thereby achieving the purpose of removing reflection.

[0047] The generated image discriminant network calculates the deviation between the generated network predicted image and the real non-reflection image sample through the following loss function:

[0048] L(θ)=ω1L feat (θ)+ω2L adv (θ)+ω3L excl (θ)

[0049] Where: L(θ) is the total loss function, L feat (θ) is the feature loss function, L adv (θ) is the adversarial loss function, L excl (θ) is the exclusion loss function, θ represents the network weight, ω1, ω2, and ω3 are weighting parameters, with values ​​of ω1 = 0.1, ω2 = 0.01, and ω3 = 1.0, respectively; the three loss functions are linearly combined as the total network loss function.

[0050] The server used in this method experiment was equipped with an AMD Ryzen 9 5950x 16-core 3.40GHz CPU and two NVIDIA GeForce RTX 3090 24GB GPUs, running Ubuntu 20.04. The network was based on the TensorFlow framework, and a pre-trained VGG-19 network was used for input image feature extraction. Hypercolumn features were composed of the VGG-19's conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2 feature layers, along with the input features. The training set consists of three parts: ① The image set obtained from Flickr (Flickr dataset), totaling 27,400 images, half of which are used as reflection layer images and the other half as transmission layer images for synthesizing reflection images; ② The real shooting dataset provided by Base Net (Berkeley real dataset), which contains 109 pairs of images in a group consisting of mixed images with glass reflection and transmission images without reflection; ③ The real windshield dataset taken in this paper (Our real dataset), which contains 24 sets of windshield reflection images and transmission images without reflection at different angles, and is randomly cropped (Crop) to enhance it into 120 sets of image pairs of different scales.

[0051] In the experiment, 5,000 pairs of images were randomly selected from the Flickr image collection as the synthetic training set, 90 pairs were selected from the real dataset provided by BaseNet, and 100 pairs of images were selected from the windshield dataset captured in this paper as the real training set. To test the network's performance, three test sets were selected for quantitative evaluation, using the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) metrics. The three test sets were: ① a synthetic dataset provided by CEILNet, containing 123 pairs of synthetic image pairs, denoted as Synthetic; ② the remaining 19 pairs of images from the real dataset provided by BaseNet, denoted as Real; and ③ the remaining 20 pairs of images from the windshield dataset captured and enhanced in this paper, denoted as Our Real.

[0052] The training and testing processes are as follows: pairs of training set samples are fed into the network. Specifically, the reflection-bearing image is used as input, and the non-reflection image is fed into the generative network after generating the de-reflection prediction image. The two images are then fed into the loss function. After the network iterates and converges, the network weights are solidified, completing the network model training. The network is then used to predict each sample in the test set, generating quantitative evaluation results. The specific experimental parameters are shown in Table 1.

[0053] Table 1 Improved network training parameters with CBAM

[0054]

[0055] Example 1: Comparative test of CBAM submodule connection methods

[0056] This example tests the impact of the three CBAM submodule connection methods on the network reflection elimination effect and compares the reflection elimination effect with the original network Base Net. The experiment was conducted on three types of test sets. The experimental results are shown in Table 2.

[0057] Table 2 Comparative test results of CBAM submodule connection methods

[0058]

[0059]

[0060] Table 2 shows that the improved network with CBAM achieves superior results in reflection cancellation compared to the original network. Among the three submodule connection methods, the order of improvement is "channel + spatial" > "spatial + channel" > "channel||spatial". The series connection of the channel attention and spatial attention submodules significantly improves reflection cancellation compared to the parallel connection of the two submodules. The "channel + spatial" connection, which achieved the greatest improvement, increased the SSIM by 0.0082 and the PSNR by 0.6182dB in the synthetic test set, by 0.0167 and the PSNR by 0.5017dB in the Base Net real test set, and by 0.0115 and the PSNR by 0.5036dB in the test set captured in this paper. After calculation, the improved network that introduces the "channel + space" CBAM connection method has an overall performance in the test set compared to the original network Base Net: SSIM increased by 1.172%, PSNR increased by 2.755%, verifying the effectiveness of the CBAM mechanism introduced in this paper and indicating that the "channel + space" sub-module connection method can achieve the best de-reflection effect.

[0061] Example 2: Multi-level CBAM residual structure ablation experiment

[0062] To address the shallow depth of the Base Net deep convolutional network, the CBAM residual block structure is used to replace the convolutional layers in the original network, and the network depth is increased by adjusting the number of CBAM ResBlocks. Different numbers of residual blocks are stacked after the original 5 layers of CBAMResBlock, bringing the total number of CBAM ResBlocks to N layers. The CBAM submodule connection method is set to "channel + spatial," and the convolutional expansion rate is no longer set for CBAM ResBlocks after the 5th layer. Calculations show that the number of network parameters increases by 0.21% with each additional CBAM ResBlock layer, a subtle change in the number of parameters. To achieve a more significant change in the overall network performance, after comprehensively considering the relationship between the change in the number of layers and the efficiency of the ablation experiment, this paper continues to test three networks with N=9, N=13, and N=17, with an interval of 4 layers. The ablation test results are shown in Table 3.

[0063] Table 3 Ablation experimental results of multi-level CBAM residual structure

[0064]

[0065]

[0066] Table 3 shows the effect of increasing the depth of the CBAM ResBlock structure on reflection removal. The 9-layer CBAMResBlock structure achieved an overall SSIM improvement of 0.722% and a PSNR improvement of 0.802% compared to the 5-layer structure on the test set. The 13-layer structure achieved an SSIM improvement of 0.435% and a PSNR improvement of 0.124% compared to the 9-layer structure. The 17-layer structure achieved a SSIM decrease of 0.017% and a PSNR decrease of 0.043% compared to the 13-layer structure. The data shows that increasing the depth of the CBAM ResBlock from 13 to 17 layers slightly improves the SSIM on the synthetic test set and the PSNR on the real dataset, but overall performance decreases on all three test sets. Therefore, further deepening the network layers is not very effective, as it will not only lead to redundant training parameters and make training more difficult, but also increase the risk of network overfitting while consuming more resources. Therefore, the improved Base Net network using 13-layer CBAM ResBlock proposed in this paper achieves the best processing effect as a single image reflection removal method.

Claims

1. A system for eliminating reflected images from automobile glass, characterized in that: include: Image preprocessing network, transmission image generation network and generated image discrimination network, where: The image preprocessing network extracts features from the input image to obtain hypercolumn feature image data; The transmission image generation network performs deep processing on the hypercolumn feature image to generate a de-reflected prediction image; wherein: the transmission image generation network includes a first convolution module, a first attention module, a second attention residual module and a second convolution module; The first attention module is a convolutional network embedded with 5 layers of residual units; the expansion rate of the first layer of the convolutional network is 8, the expansion rate of the second layer is 16, the expansion rate of the third layer is 32, the expansion rate of the fourth layer is 64, and the expansion rate of the fifth layer is 128; The second attention module is a convolutional network embedded with 8 layers of residual units; the expansion rate of each layer of the convolutional network is 1; The residual unit includes a weight calculation part and an attention operation part; the weight calculation part is composed of a convolution layer and an activation function; the attention operation part is connected in series using channel attention and spatial attention; The generated image discrimination network sends the predicted image and the real non-reflection image samples corresponding to the input image into the multivariate loss function for discrimination, and eliminates the automobile glass reflection image system through continuous iterative training.

2. The system for eliminating automobile glass reflection images according to claim 1, characterized in that: The image preprocessing network adopts the VGG-19 convolutional neural network, and the convolutional neural network structure includes conv1_2, conv2_2, conv3_2, conv4_2 and conv5_2 feature layers.

3. The system for eliminating automobile glass reflection images according to claim 1, characterized in that: The generated image discriminant network calculates the deviation between the generated network predicted image and the real non-reflection image sample through the following multivariate loss function: L(θ)=ω1L feat (θ)+ω2L adv (θ)+ω3L excl (i) Where: L(θ) is the total loss function, L feat (θ) is the feature loss function, L adv (θ) is the adversarial loss function, L excl (θ) is the exclusion loss function, θ represents the network weight, ω1, ω2, and ω3 are weighting parameters, with values ​​of ω1 = 0.1, ω2 = 0.01, and ω3 = 1.0, respectively; the three loss functions are linearly combined as the total network loss function.

4. A method for eliminating reflection images from automobile glass, characterized in that: The method comprises the following steps: extracting features from an input image to obtain hypercolumn feature image data; A fully convolutional network with an attention residual model is used to perform deep processing on the hypercolumn feature image to generate a de-reflected predicted image; wherein: the attention residual model is composed of a first attention module and a second attention residual module; The first attention module is a convolutional network embedded with 5 layers of residual units; the expansion rate of the first layer of the convolutional network is 8, the expansion rate of the second layer is 16, the expansion rate of the third layer is 32, the expansion rate of the fourth layer is 64, and the expansion rate of the fifth layer is 128; The second attention module is a convolutional network embedded with 8 layers of residual units; the expansion rate of each layer of the convolutional network is 1; The residual unit includes a weight calculation part and an attention operation part; the weight calculation part is composed of a convolution layer and an activation function; the attention operation part is connected in series using channel attention and spatial attention; The predicted image and the real non-reflection image sample corresponding to the input image are fed into the multivariate loss function for discrimination, and the generative network model is converged through continuous iteration, where: The deviation between the network prediction image and the real non-reflection image sample is calculated by the following multivariate loss function: L(θ)=ω1L feat (θ)+ω2L adv (θ)+ω3L excl (i) Where: L(θ) is the total loss function, L feat (θ) is the feature loss function, L adv (θ) is the adversarial loss function, L excl (θ) is the exclusion loss function, θ represents the network weight, ω1, ω2, and ω3 are weighting parameters, with values ​​of ω1 = 0.1, ω2 = 0.01, and ω3 = 1.0, respectively; the three loss functions are linearly combined as the total network loss function.

Images