Reflection interference removing method and system based on reflection area detection and edge attention

Abstract

The invention belongs to the technical field of computers, and particularly relates to a reflection interference removing method and system based on reflection area detection and edge attention, and the method comprises the steps: employing the idea of a generative adversarial network on a main body, enabling a mixed image containing reflection interference to serve as the input of a model after preprocessing; inputting an input mixed image containing reflection interference into a reflection region detection network module, and obtaining multi-scale Gaussian-Laplacian features of the image through a reflection feature enhancement network module, and finally participating in synthesis of image gradient features of enhanced reflection; the obtained image gradient features serve as the input of the whole generation network, the features are extracted through the shallow feature extraction network, the extracted features are transmitted to the discrimination network in a jump connection mode, the output image after reflection interference removal is obtained, meanwhile, the quality of the generated image after reflection interference removal is discriminated, and the output image after reflection interference removal is obtained. And guiding the generation of the generator in the next round until the output quality meets the requirement. According to the invention, the influence of reflection interference on the image can be eliminated in a complex light external scene, and the quality of the license image acquired by the system is improved, so that the license image meets the image quality requirements of other downstream tasks.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the field of computer technology, and in particular relates to a method and system for removing reflection interference based on reflection area detection and edge attention. Background Art

[0002] In the actual certificate image acquisition system, in order to ensure the imaging quality, we introduced structures such as reflectors and glass objective lenses. The introduction of reflectors and glass objective lenses will more or less bring about the problem of reflection interference in the collected images. Reflection interference refers to the phenomenon that the external scene is reflected through the optical structures such as objective glass and reflectors under the external ambient light, resulting in glare, highlights and artifacts in the captured certificate image, which interferes with image recognition. The main reasons for the reflection interference in the images collected by the system are as follows: on the one hand, the multi-spectral light of the certificate image acquisition device, especially the visible light, is not highly penetrable. When passing through the thick objective glass, the light is refracted on the glass surface, forming a blurred double image; on the other hand, when the external ambient light is strong, the external scene will be imaged on the reflector through the objective glass, and a mirror reflection image of the external scene will be formed on the surface of the objective glass. Although the imaging effect of the system can be intelligently adjusted when the ambient light is strong or weak by adjusting the exposure time of the camera, when the external ambient light is too strong, the reflection interference cannot be suppressed by adjusting the exposure time, and the overall imaging quality of the system will be reduced.

[0003] In engineering practice, it is difficult and ineffective to use non-deep learning methods to remove reflective interference from certificate images: First, during the operation of the certificate image acquisition system, only a single image is obtained under each spectrum. The single image de-reflection algorithm usually requires additional user-calibrated prior conditions to assist in the de-reflection interference task, which is time-consuming and labor-intensive, and the prior conditions may fail in different types of reflection scenes; second, due to the uncertainty of ambient lighting and external background, the scale of the reflective area in the single certificate image acquired by the system varies greatly and is difficult to predict, the reflection intensity of some areas varies, the edges of the reflective area are blurred, and it is difficult to locate the reflective area; third, when the external ambient light is too strong, artifacts and other image interference that seriously affect the reading of the certificate will appear in the captured certificate image, suppressing the expression of image edge features in the certificate image, making image recovery more difficult.

[0004] Through the above analysis, the problems and defects of the existing technology are: it is difficult to avoid the reflection problem in image imaging with the existing methods, and it is known that the existing traditional image processing methods using non-deep learning have weak ability to remove reflection interference, have great limitations, poor effects, and are difficult. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method and system for removing reflection interference based on reflection area detection and edge attention.

[0006] The present invention is implemented as follows: a method for removing reflection interference based on reflection area detection and edge attention, comprising the following steps:

[0007] S1: The idea of ​​generative adversarial network is adopted in the main body, and the mixed image containing reflection interference is preprocessed as the input of the model;

[0008] S2: The mixed image containing reflection interference input in S1 is input into the reflection area detection network module, and then passes through the reflection feature enhancement network module to obtain the multi-scale Gaussian-Laplacian features of the image and finally participate in the synthesis of the image gradient features of the enhanced reflection;

[0009] S3: The image gradient features obtained in S2 are used as the input of the entire generation network, and the features are extracted through the shallow feature extraction network;

[0010] S4: The features extracted in S3 are transmitted to the discriminant network via skip connections. While obtaining the output image after de-reflection interference, the quality of the generated image after de-reflection interference is judged to guide the generation of the next round of generators until the output quality meets the requirements.

[0011] Further, the S1 specifically includes:

[0012] The task of removing image reflection interference usually considers that the image containing reflection interference is a mixture of the transmission image and the reflection image. When the traditional method of calculating the mixture ratio of the two through some image features or using a mixing matrix to separate them is used, it has great limitations in processing the task of removing reflection interference from the license image, which is specifically characterized by the inability to process images with uneven reflection intensity and poor generalization ability. Therefore, an image reflection interference removal algorithm based on deep learning is used to achieve this.

[0013] Since the generative adversarial network can effectively enhance image details in the generation stage and can also guide the image generation effect by adjusting the attention distribution, improvements are made based on the generative adversarial network.

[0014] The mixed images with reflection interference collected in the license acquisition hardware system are resized and instance normalized to ensure that the images input to the network are of the same size and have no influence on each other; instance normalization refers to the process of calculating the mean and standard deviation of each input feature map on a single channel, and then using the mean and standard deviation on each channel to normalize the elements in the single channel; instance normalization has great advantages in image style transfer. For a feature map x with a batch size of N, a length, width, and number of channels of h, w, and c, respectively, its specific mathematical definition is as follows:

[0015]

[0016] where x hw represents an image of scale h×w under a single channel, μ(x) and σ(x) are the mean and variance respectively, γ and β are variables learned during training, and ε is a small constant.

[0017] Further, the S2 specifically includes:

[0018] According to the characteristic that the gradient of the reflected image in the mixed image is sparser than that of the transmitted image, a reflection area detection network is designed. The mixed image passes through the multi-scale reflection area detection network to obtain the multi-scale Gaussian-Laplacian features of the image, and then further refines the reflection features through the reflection feature enhancement network that integrates channel attention, outputs the reflection area probability map and the reflection image prediction map, and then guides the image generation of the next stage by suppressing the reflection dominant area and enhancing the transmission dominant area;

[0019] According to the phenomenon that the edge features of the projected image are usually more obvious than those of the reflected image, a reflection feature enhancement network based on edge attention is designed to enhance the edge of the transmitted image; the reflection feature enhancement module has two parts of input, one of which is the multi-scale Gaussian-Laplacian feature I of the original image obtained in the reflection area detection module out , and the second is the output I of the image completely passing through the reflection area detection module RD , I out The feature enhancement is performed through four cascaded squeeze-excitation residual modules and then combined with I RD The weighted product obtains the image gradient feature of the enhanced reflection;

[0020] Removing reflection interference can be regarded as the task of separating the reflection image and the transmission image in the mixed image. In the traditional algorithm of removing interference by using image transformation, the selected prior conditions are highly dependent on the scene. The prior conditions may fail in a specific scene, which directly affects the effect of reflection removal.

[0021] In order to ensure the acquisition of multi-scale image gradient features, a multi-scale feature extraction module is used in the network structure. The input image is scaled by multi-scale global pooling, and the image size is scaled to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original scale. The original scale image and the multi-scale image obtained by the pooling operation are respectively subjected to Gaussian-Laplacian convolution to obtain multi-scale image gradient features.

[0022] Laplace convolution is a common edge detection method. It obtains the edge features of the image by performing second-order differential on the image, but it is sensitive to image noise and discrete points. In order to overcome this problem, Gaussian smoothing is usually performed before Laplace convolution. The convolution kernel is calculated as follows:

[0023]

[0024] where σ 2 is the standard deviation, and its selection is positively correlated with the image smoothing effect. 2 Fine-tune the value;

[0025] The obtained multi-scale Gaussian-Laplacian feature maps are upsampled to the original scale size and spliced ​​using the bilinear interpolation algorithm; To represent the image of the corresponding scale after the downsampling operation, use Represents the upsampled image of the corresponding scale. The output image is:

[0026]

[0027] Use 3 squeeze-excitation residual modules to enhance the output image I out The multi-scale Gaussian-Laplacian features of the image are obtained by the squeeze-excitation residual module through the two steps of squeeze and excitation to model the interdependence between the channels of the image. The squeeze is performed by performing global average pooling on the feature map to obtain the global compressed feature vector of the current feature map. For the input image X∈H×W×C, the squeeze step can be written as the following formula:

[0028]

[0029] Where Z c refers to the output related to the cth channel, input x c From the convolutional layer; the excitation is to obtain the weight of each channel in the feature map through two fully connected layers, activate it through the Sigmoid function, and then use the weighted feature map as the input of the next layer of the network. The excitation step can be written as the following formula:

[0030]

[0031] T1 and T2 are two linear layers, and finally the feature strength of each channel is adaptively adjusted through the global loss function of the network;

[0032] After obtaining the enhanced image features, the image features are activated by the PReLU function. The PReLU function formula can be written as:

[0033]

[0034] where y i is the input of the ith channel, a i Control the slope of the negative semi-axis, the initial value is a i = 0.2, its value can be adaptively updated through back propagation in the subsequent training process, and the process of feature enhancement and activation is recorded as f LOG Finally, through the 3×3 convolution kernel, the Sigmoid activation normalization is used to obtain the final reflection area detection probability map I RD , I RD can be expressed as:

[0035] I RD =Sigmoid(Conv(f LOG (I out )))

[0036] In this way, the mixed image can obtain preliminary reflection image gradient features through the reflection area detection module, which provides basic clues for predicting the reflection image. In order to better predict the reflection image, the reflection feature enhancement module is used to further refine the gradient features of the reflection image. The reflection feature enhancement module group has two parts of input, one of which is the original image obtained by the multi-scale Gaussian-Laplacian feature extraction module. out , and the second is the final output I of the reflection area detection module RD , feature enhancement is performed through four cascaded squeeze-excitation residual modules, and then combined with I RD Weighted product is used to further enhance the gradient features of the reflected image;

[0037] In order to enhance the edge representation ability of the network, an edge attention module is introduced into the network. In traditional edge detection schemes, only the binary contour of the image is usually extracted. However, in the task of removing reflection interference, obtaining the binary edge contour is not effective for the subsequent restoration of the transmission image, because in the strong reflection area, there may be indistinguishable hard boundaries that interfere with the edge detection of the actual transmission image. Therefore, the average absolute color difference method is chosen to obtain the image edge representation. The color difference between a certain pixel and the surrounding pixels is calculated in the three RGB channels of the input image and the weighted average is calculated. The image edge is estimated based on the average absolute color difference. The specific calculation method is as follows:

[0038]

[0039] Where (x, y) is the coordinate of the corresponding pixel in the image, and c refers to the RGB color space channel of the image. The edge map of the mixed image obtained in this way is more effective in suppressing the hard boundaries of the reflective area than the binary edge map obtained by graying the image, because the color gamut of the hard boundaries of the reflective area is usually relatively single and has little impact on the mean absolute color difference. The obtained edge map is introduced into the network as an additional channel input to increase the edge attention of the network.

[0040] Further, the S3 specifically includes:

[0041] The generative network consists of a convolutional layer, a ReLU activation layer, an instance normalization layer, and a coordinated attention layer. The generative network extracts features from the input image. To enhance the image feature extraction capability, a coordinated attention module is introduced into the generative network. This module aggregates the features in the vertical and horizontal directions into two separate feature maps by performing pooling operations in the row and column dimensions. The acquired feature maps are encoded into two attention maps, each of which can capture long-distance dependencies in the row or column spatial direction. This attention mechanism can efficiently integrate the global features of the image, not only to obtain cross-channel feature information, but also to obtain coordinate-sensitive information, which helps the model identify and locate features of interest. In the feature extraction link of the mixed image, the coordinate positioning of the reflection area is an important basis for removing reflection interference. Channel attention solves the problem of feature acquisition, and spatial attention solves the problem of feature positioning. Coordinated attention provides the network with both spatial attention and channel attention, while only increasing a small amount of computational complexity, so that the generative network can efficiently obtain the features of the transmission image.

[0042] After obtaining the separated and enhanced transmission image features through S2, it is used as the input of the generative adversarial network for subsequent feature reconstruction and output; the generative adversarial network is a generative model that combines the characteristics of the generative model and the discriminative model. The network consists of a generator network and a discriminator network. The generator is responsible for generating new data samples according to the characteristic distribution of the sample data, and the discriminator gives feedback to further optimize the generator. The two neural networks learn by playing against each other, and finally guide the data samples generated by the generator to be as close to the real data samples as possible; the network can be used in an unsupervised or semi-supervised manner, which greatly reduces the requirements for sample data;

[0043] The basic process of generating adversarial networks is that the generator network G learns the random input z of the model, the generator outputs the generated sample G(z), and then inputs the generated sample and the real sample G(x) into the discriminator. The discriminator marks the real sample as 1 and the generated sample as 0. The purpose is to train a maximum discriminator D to classify the sample accurately, and on this basis, minimize the adversarial loss log(1-D(G(z))) to optimize the generator G, thereby reducing the accuracy of the discriminator classification. The valuation functions of the two are completely opposite, and a zero-sum game is performed according to the minimization and maximization valuation function V(G,D), which can be written as:

[0044]

[0045] Where x represents the real sample data, p data (x) represents the true sample data distribution, z represents random input data, p z (z) represents the prior distribution of random input data; when p z =p data When , there is an optimal solution to the zero-sum game.

[0046] Further, the S4 specifically includes:

[0047] The main function of the discriminant network is to expand the feature vector of the acquired feature map, and finally obtain a generated image with the same scale as the input image; in addition, jump connections are introduced in the generation network and the discriminant network, which can directly transfer image data from the generation network to the discriminant network to reduce the loss of image detail information; in addition, a feature converter composed of 6 cascaded residual modules is added between the generation network and the discriminant network. Its main function is to convert the image features extracted by the discriminant network into the corresponding feature map, and it can effectively alleviate the problem of gradient disappearance during training;

[0048] The decoder consists of a deconvolution layer, a ReLU activation layer, a normalization layer, a Tanh activation layer, etc. The main function of the decoder is to expand the feature vectors obtained from the feature converter and the jump connection to obtain a generated image with the same scale as the input image. At the same time, it participates in the discrimination process of the entire generative adversarial network, and improves the refinement and generation quality of the entire network through the adversarial learning process.

[0049] Another object of the present invention is to provide a de-reflection interference system based on reflection area detection and edge attention to implement the de-reflection interference method based on reflection area detection and edge attention, the system includes an image preprocessing module, a reflection area detection network module, a reflection feature enhancement network module, a generative adversarial network module including a generation network module and a discrimination network module

[0050] The image preprocessing module is used to normalize the size of the image after it is input into the network;

[0051] A reflection area detection network module is connected to the image preprocessing module and is used to obtain multi-scale Gaussian-Laplace features of the image in the reflection area detection network;

[0052] The reflection feature enhancement network module is connected to the reflection area detection network module, and is used to further refine the reflection features through the reflection feature enhancement network that integrates channel attention, and output the probability map of the reflection area and the prediction map of the reflection image to the generative adversarial network;

[0053] The generative adversarial network module includes a generative network module and a discriminative network module, which is used to guide image generation by suppressing the reflection-dominated area and enhancing the transmission-dominated area under the joint action of the generative network and the discriminative network.

[0054] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the de-reflection interference method based on reflection area detection and edge attention.

[0055] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the de-reflection interference method based on reflection area detection and edge attention.

[0056] Another object of the present invention is to provide a certificate image data processing terminal, which is used to implement the certificate image acquisition system of the de-reflection interference method based on reflective area detection and edge attention.

[0057] Another object of the present invention is to provide a cross-platform computer application that can be deployed on Windows, Linux and most domestic operating systems, and is used to complete the steps of the de-reflection interference method based on reflection area detection and edge attention in combination with specific hardware.

[0058] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0059] First, (1) A de-reflection interference method based on reflection area detection and edge attention of the present invention can be used for image de-reflection interference work of a certificate acquisition system in complex lighting scenes. According to the characteristics of the gradient difference between the reflection-dominated area and the transmission-dominated area in a mixed image containing reflection interference, reflection area detection is introduced into the image generation network to adjust the network's attention to specific areas; and because of the problems of blurred text and unclear details caused by unclear edge features in the mixed image, an edge attention module is introduced into the network to retain as much image detail edge information as possible.

[0060] Compared with similar algorithms, the algorithm in this chapter is superior to similar algorithms in terms of model size, running speed and technical indicators.

[0061] (2) The present invention proposes a method for removing reflection interference based on reflection area detection and edge attention, which can provide a theoretical basis and strong technical support for the development of downstream work to a great extent. For a long time, many of the certificate acquisition systems currently in service have only completed basic image acquisition tasks. If you want to further analyze the images collected by the certificate equipment in detail, you need to use solutions or equipment other than the certificate acquisition system. There are two main reasons for this phenomenon. The first reason is that this type of certificate acquisition system does not have the ability to adaptively resist reflection interference. At the same time, it is also more or less affected by external light sources and has the problem of reflection interference, which makes it difficult or even impossible to perform a series of downstream tasks such as machine-readable code recognition, full-width recognition and chip reading on the collected certificate images under visible light; the second reason is that the currently feasible removal of reflection interference algorithms based on traditional image processing methods are extremely dependent on the use scenario. If they are not suitable for the environment in which the certificate acquisition system is located, it will cause the processed image to have poor effect or too long processing time, and fail to meet the system parameter requirements.

[0062] The de-reflection interference method based on reflective area detection and edge attention proposed in the present invention adopts an end-to-end structural network based on a generative adversarial network, which is relatively easy to deploy as a module embedded in the system, has little impact on the structure of the entire certificate collection system, and is easy to deploy; at the same time, since a deep learning framework is used to complete the task of de-reflection interference, the module parameters can be trained before being deployed to the system, which makes the time overhead of actually putting the technical solution into production less than that of traditional image processing methods, thereby saving more time for downstream tasks and enabling the certificate collection system to complete more personalized and modular collection and reading of the certificates to be inspected within a limited time.

[0063] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0064] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0065] The expected benefits and commercial value of the transformation of this technical solution are considerable. First, this technical solution solves the problem of glare, bright spots and other reflective interference in the certificate images collected by the certificate collection equipment under complex lighting conditions. This problem greatly improves the phenomenon that the recognition rate and the success rate of counterfeiting detection in downstream tasks such as full-frame reading and authentication of certificates are reduced due to the reflection interference in the collected images, so that the certificate collection equipment can integrate image acquisition, reading and authentication, and achieve the good effect of one machine for multiple uses and one machine for reuse. This will attract airports, border inspections, customs and various enterprises and institutions with relevant needs to choose the all-in-one equipment after the technical solution is converted with its own quality and advantages, instead of having to purchase equipment with different functions many times to complete the corresponding combination tasks, and truly realize the integration and intelligence of certificate collection equipment, and the functions of the collection equipment are connected and streamlined. Secondly, this technical solution provides a new idea for solving this kind of reflection interference problem. This technical solution can be applied to different scenarios, not just limited to the certificate collection system. It has great application potential and is expected to be adopted and accepted by other industries, so as to obtain continuous benefits. In addition, the patent protection of the technical solution will further ensure competitive advantages and market share, bringing long-term business value and competitive advantages to enterprises. In summary, the expected conversion benefits and commercial value of this technical solution are high, and it is expected to become an important technical and commercial standard for the certificate collection system of airports and border inspection ports.

[0066] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0067] For a certificate acquisition system, most systems currently use traditional image processing methods to deal with the phenomenon that the collected images will produce reflection interference under certain light or certain types of lighting conditions. Some systems themselves do not even have the function of adaptively removing reflection interference. When facing complex external light conditions, traditional image processing methods may only have significant effects on the anti-reflection interference phenomenon of one or one type of external light, and do not have generalization capabilities. The method proposed in this technical solution is based on reflection area detection and reflection edge attention. In practical applications, it can adaptively process various reflection phenomena generated under different light source conditions, and has good generalization in use. Prior to this, there have been no deep learning-based methods and solutions that can robustly solve the anti-reflection interference of certificate acquisition systems at home and abroad. In addition, the present invention has made important breakthroughs in key technologies such as feature extraction and feature fusion for this type of problem, laying a solid foundation for further research on the removal of reflection interference in certificate acquisition systems and more downstream tasks and applications based on image quality and image integrity after removal of reflection interference. Therefore, the present invention fills the technical gaps in feature extraction and fusion and removal of reflection interference in the industry at home and abroad, and brings new technical solutions and development directions for the development of certificate acquisition systems.

[0068] Fourth, the technical progress of the de-reflection interference method based on reflection area detection and edge attention is mainly reflected in the following aspects:

[0069] 1. The introduction of generative deep neural network improves the generalization of the model: This method is based on generative adversarial network, which solves the problem of proportional limitation when using traditional image processing methods to solve this type of de-reflection interference task, such as calculating the mixing ratio of reflected image and transmitted image through some image features or using mixing matrix for separation, thereby improving the robustness and generalization of the entire model.

[0070] 2. Construct a reflection area detection module: The reflection area detection module obtains image gradient features by learning multi-scale Gaussian-Laplacian features in the mixed image, detects and locates the area dominated by the reflection image in the mixed image, and obtains a reflection interference probability map, which is used to indicate the probability that a certain area in the image is dominated by the reflection image. The establishment of this reflection area detection module can effectively filter out local noise caused by reflection interference. Compared with the weakness of traditional image processing methods that the prior conditions may fail in specific scenarios, this method has a better effect on reflection removal.

[0071] 3. Introduce a reflection feature enhancement network module based on edge attention: This method uses the mean absolute color difference method to obtain image edge representation. The color difference between a certain pixel and the surrounding pixels is calculated and the weighted average is obtained under the RGB three channels of the input image, and the image edge is estimated based on the mean absolute color difference. The edge map of the mixed image obtained in this way has a more obvious suppression effect on the hard boundary of the reflection area than the binary edge map obtained by graying the image. The obtained edge map is introduced into the network as an additional channel input to increase the edge attention of the network and improve the feature extraction ability of the network.

[0072] 4. Use multi-dimensional loss function to improve the output image quality: The loss function of the de-reflection interference network mainly comes from two aspects. On the one hand, it is the pixel loss between the generated transmission image and the reference transmission image. On the other hand, it is the perceptual loss and adversarial loss from the network. The pixel loss can optimize the quality of the generated image and make the generated image obtain a higher peak signal-to-noise ratio. The adversarial loss helps to improve the intuitive effect of the network output image. Overall, the weighted average of multiple loss functions can improve the quality of the generated image.

[0073] The de-reflection interference method based on reflection area detection and edge attention has made significant technical progress in network construction, reflection area detection, reflection feature enhancement, and output image quality improvement. These advances not only improve the accuracy and reliability of the task, but also provide new ideas and more effective technical means for deep learning methods to solve the de-reflection problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of a method for removing reflection interference based on reflection area detection and edge attention provided by an embodiment of the present invention;

[0075] Figure 2 It is a structural diagram of a certificate image acquisition system of a method for removing reflection interference based on reflection area detection and edge attention provided by an embodiment of the present invention;

[0076] Figure 3 is a schematic diagram of a network implementation of a reflection area detection module provided in an embodiment of the present invention;

[0077] Figure 4 It is a schematic diagram of a network implementation of an edge attention reflection feature enhancement module provided in an embodiment of the present invention;

[0078] Figure 5 Schematic diagram of an implementation of an image generation network based on a generative adversarial network provided by an embodiment of the present invention;

[0079] Figure 6It is the experimental result of the whole network provided by the embodiment of the present invention, including the ablation experiment inside the network and the comparative experiment with similar algorithms. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0081] like Figure 1 As shown, an embodiment of the present invention provides a method for removing reflection interference based on reflection area detection and edge attention, comprising the following steps:

[0082] S1: The idea of ​​generative adversarial network is adopted in the main body, and the mixed image containing reflection interference is preprocessed as the input of the model;

[0083] S2: The mixed image containing reflection interference input in S1 is input into the reflection area detection network module, and then passes through the reflection feature enhancement network module to obtain the multi-scale Gaussian-Laplacian features of the image and finally participate in the synthesis of the image gradient features of the enhanced reflection;

[0084] S3: The image gradient features obtained in S2 are used as the input of the entire generation network, and the features are extracted through the shallow feature extraction network;

[0085] S4: The features extracted in S3 are transmitted to the discriminant network via skip connections. While obtaining the output image after de-reflection interference, the quality of the generated image after de-reflection interference is judged to guide the generation of the next round of generators until the output quality meets the requirements.

[0086] The S1 specifically includes:

[0087] The task of removing image reflection interference usually considers that the image containing reflection interference is a mixture of the transmission image and the reflection image. When the traditional method of calculating the mixture ratio of the two through some image features or using a mixing matrix to separate them is used, it has great limitations in processing the task of removing reflection interference from the license image, which is specifically characterized by the inability to process images with uneven reflection intensity and poor generalization ability. Therefore, an image reflection interference removal algorithm based on deep learning is used to achieve this.

[0088] Since the generative adversarial network can effectively enhance image details in the generation stage and can also guide the image generation effect by adjusting the attention distribution, improvements are made based on the generative adversarial network.

[0089] The mixed images with reflection interference collected in the license acquisition hardware system are resized and instance normalized to ensure that the images input to the network are of the same size and have no influence on each other; instance normalization refers to the process of calculating the mean and standard deviation of each input feature map on a single channel, and then using the mean and standard deviation on each channel to normalize the elements in the single channel; instance normalization has great advantages in image style transfer. For a feature map x with a batch size of N, a length, width, and number of channels of h, w, and c, respectively, its specific mathematical definition is as follows:

[0090]

[0091]

[0092] where x hw represents an image of scale h×w under a single channel, μ(x) and σ(x) are the mean and variance respectively, γ and β are variables learned during training, and ε is a small constant.

[0093] Further, the S2 specifically includes:

[0094] According to the characteristic that the gradient of the reflected image in the mixed image is sparser than that of the transmitted image, a reflection area detection network is designed. The mixed image passes through the multi-scale reflection area detection network to obtain the multi-scale Gaussian-Laplacian features of the image, and then further refines the reflection features through the reflection feature enhancement network that integrates channel attention, outputs the reflection area probability map and the reflection image prediction map, and then guides the image generation of the next stage by suppressing the reflection dominant area and enhancing the transmission dominant area;

[0095] According to the phenomenon that the edge features of the projected image are usually more obvious than those of the reflected image, a reflection feature enhancement network based on edge attention is designed to enhance the edge of the transmitted image; the reflection feature enhancement module has two parts of input, one of which is the multi-scale Gaussian-Laplacian feature I of the original image obtained in the reflection area detection module out , and the second is the output I of the image completely passing through the reflection area detection module RD , I out The feature enhancement is performed through four cascaded squeeze-excitation residual modules and then combined with I RD The weighted product obtains the image gradient feature of the enhanced reflection;

[0096] Removing reflection interference can be regarded as the task of separating the reflection image and the transmission image in the mixed image. In the traditional algorithm of removing interference by using image transformation, the selected prior conditions are highly dependent on the scene. The prior conditions may fail in a specific scene, which directly affects the effect of reflection removal.

[0097] In order to ensure the acquisition of multi-scale image gradient features, a multi-scale feature extraction module is used in the network structure. The input image is scaled by multi-scale global pooling, and the image size is scaled to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original scale. The original scale image and the multi-scale image obtained by the pooling operation are respectively subjected to Gaussian-Laplacian convolution to obtain multi-scale image gradient features.

[0098] Laplace convolution is a common edge detection method. It obtains the edge features of the image by performing second-order differential on the image, but it is sensitive to image noise and discrete points. In order to overcome this problem, Gaussian smoothing is usually performed before Laplace convolution. The convolution kernel is calculated as follows:

[0099]

[0100] where σ 2 is the standard deviation, and its selection is positively correlated with the image smoothing effect. 2 Fine-tune the value;

[0101] The obtained multi-scale Gaussian-Laplacian feature maps are upsampled to the original scale size and spliced ​​using the bilinear interpolation algorithm; To represent the image of the corresponding scale after the downsampling operation, use Represents the upsampled image of the corresponding scale. The output image is:

[0102]

[0103] Use 3 squeeze-excitation residual modules to enhance the output image I out The multi-scale Gaussian-Laplacian features of the image are obtained by the squeeze-excitation residual module through the two steps of squeeze and excitation to model the interdependence between the channels of the image. The squeeze is performed by performing global average pooling on the feature map to obtain the global compressed feature vector of the current feature map. For the input image X∈H×W×C, the squeeze step can be written as the following formula:

[0104]

[0105] Where Z c refers to the output related to the cth channel, input x c From the convolutional layer; the excitation is to obtain the weight of each channel in the feature map through two fully connected layers, activate it through the Sigmoid function, and then use the weighted feature map as the input of the next layer of the network. The excitation step can be written as the following formula:

[0106]

[0107] T1 and T2 are two linear layers, and finally the feature strength of each channel is adaptively adjusted through the global loss function of the network;

[0108] After obtaining the enhanced image features, the image features are activated by the PReLU function. The PReLU function formula can be written as:

[0109]

[0110] where y i is the input of the ith channel, a i Control the slope of the negative semi-axis, the initial value is a i = 0.2, its value can be adaptively updated through back propagation in the subsequent training process, and the process of feature enhancement and activation is recorded as f LOG Finally, through the 3×3 convolution kernel, the Sigmoid activation normalization is used to obtain the final reflection area detection probability map I RD , I RD can be expressed as:

[0111] I RD =Sigmoid(Conv(f LOG (I out )))

[0112] In this way, the mixed image can obtain preliminary reflection image gradient features through the reflection area detection module, which provides basic clues for predicting the reflection image. In order to better predict the reflection image, the reflection feature enhancement module is used to further refine the gradient features of the reflection image. The reflection feature enhancement module group has two parts of input, one of which is the original image obtained by the multi-scale Gaussian-Laplacian feature extraction module. out , and the second is the final output I of the reflection area detection module RD , feature enhancement is performed through four cascaded squeeze-excitation residual modules, and then combined with I RD Weighted product is used to further enhance the gradient features of the reflected image;

[0113] In order to enhance the edge representation ability of the network, an edge attention module is introduced into the network. In traditional edge detection schemes, only the binary contour of the image is usually extracted. However, in the task of removing reflection interference, obtaining the binary edge contour is not effective for the subsequent restoration of the transmission image, because in the strong reflection area, there may be indistinguishable hard boundaries that interfere with the edge detection of the actual transmission image. Therefore, the average absolute color difference method is chosen to obtain the image edge representation. The color difference between a certain pixel and the surrounding pixels is calculated in the three RGB channels of the input image and the weighted average is calculated. The image edge is estimated based on the average absolute color difference. The specific calculation method is as follows:

[0114]

[0115] Where (x, y) is the coordinate of the corresponding pixel in the image, and c refers to the RGB color space channel of the image. The edge map of the mixed image obtained in this way is more effective in suppressing the hard boundaries of the reflective area than the binary edge map obtained by graying the image, because the color gamut of the hard boundaries of the reflective area is usually relatively single and has little impact on the mean absolute color difference. The obtained edge map is introduced into the network as an additional channel input to increase the edge attention of the network.

[0116] The S3 specifically includes:

[0117] The generative network consists of a convolutional layer, a ReLU activation layer, an instance normalization layer, and a coordinated attention layer. The generative network extracts features from the input image. To enhance the image feature extraction capability, a coordinated attention module is introduced into the generative network. This module aggregates the features in the vertical and horizontal directions into two separate feature maps by performing pooling operations in the row and column dimensions. The acquired feature maps are encoded into two attention maps, each of which can capture long-distance dependencies in the row or column spatial direction. This attention mechanism can efficiently integrate the global features of the image, not only to obtain cross-channel feature information, but also to obtain coordinate-sensitive information, which helps the model identify and locate features of interest. In the feature extraction link of the mixed image, the coordinate positioning of the reflection area is an important basis for removing reflection interference. Channel attention solves the problem of feature acquisition, and spatial attention solves the problem of feature positioning. Coordinated attention provides the network with both spatial attention and channel attention, while only increasing a small amount of computational complexity, so that the generative network can efficiently obtain the features of the transmission image.

[0118] After obtaining the separated and enhanced transmission image features through S2, it is used as the input of the generative adversarial network for subsequent feature reconstruction and output; the generative adversarial network is a generative model that combines the characteristics of the generative model and the discriminative model. The network consists of a generator network and a discriminator network. The generator is responsible for generating new data samples according to the characteristic distribution of the sample data, and the discriminator gives feedback to further optimize the generator. The two neural networks learn by playing against each other, and finally guide the data samples generated by the generator to be as close to the real data samples as possible; the network can be used in an unsupervised or semi-supervised manner, which greatly reduces the requirements for sample data;

[0119] The basic process of generating adversarial networks is that the generator network G learns the random input z of the model, the generator outputs the generated sample G(z), and then inputs the generated sample and the real sample G(x) into the discriminator. The discriminator marks the real sample as 1 and the generated sample as 0. The purpose is to train a maximum discriminator D to classify the sample accurately, and on this basis, minimize the adversarial loss log(1-D(G(z))) to optimize the generator G, thereby reducing the accuracy of the discriminator classification. The valuation functions of the two are completely opposite, and a zero-sum game is performed according to the minimization and maximization valuation function V(G,D), which can be written as:

[0120]

[0121] Where x represents the real sample data, p data (x) represents the true sample data distribution, z represents random input data, p z (z) represents the prior distribution of random input data; when p z =p data When , there is an optimal solution to the zero-sum game.

[0122] The present invention adopts the idea of ​​generative adversarial network, takes the mixed image with reflection interference as input, and the network predicts the transmission image without reflection interference. The specific structure is shown in the attached figure. Figure 5 As shown. The overall architecture of the network is a generative adversarial network. According to the characteristic that the gradient of the reflected image in the mixed image is sparser than that of the transmitted image, the reflection area detection network and the reflection feature enhancement network in step S2 are designed; in addition, in some algorithms for removing reflection interference, due to the deep neural network layer and the multiple downsampling layers in the network, the quality of the final output image of the model is reduced compared with the input image, resulting in the effect of removing reflection interference being not as expected. In order to solve such a problem, a jump connection method is used between the generator and the discriminator of the generative adversarial network in the present invention to transfer the low-level features obtained by the shallow feature extraction network to the output layer network, so as to avoid the problem of gradient disappearance, accelerate the training process, and effectively improve the quality of the output image with removed reflection interference.

[0123] The S4 specifically includes:

[0124] The main function of the discriminant network is to expand the feature vector of the acquired feature map, and finally obtain a generated image with the same scale as the input image; in addition, jump connections are introduced in the generation network and the discriminant network, which can directly transfer image data from the generation network to the discriminant network to reduce the loss of image detail information; in addition, a feature converter composed of 6 cascaded residual modules is added between the generation network and the discriminant network. Its main function is to convert the image features extracted by the discriminant network into the corresponding feature map, and it can effectively alleviate the problem of gradient disappearance during training;

[0125] The decoder consists of a deconvolution layer, a ReLU activation layer, a normalization layer, a Tanh activation layer, etc. The main function of the decoder is to expand the feature vectors obtained from the feature converter and the jump connection to obtain a generated image with the same scale as the input image. At the same time, it participates in the discrimination process of the entire generative adversarial network, and improves the refinement and generation quality of the entire network through the adversarial learning process.

[0126] The detailed image and intermediate data processing of the de-reflection interference method based on reflection area detection and edge attention involves multiple steps, aiming to improve the overall quality of the image after de-reflection interference by learning the separation process of the reflection image and the projected image and the key feature extraction and fusion of the reflection image through deep learning methods. The following is a detailed description of each step of the method:

[0127] (1) Data input and preprocessing

[0128] Image data preprocessing:

[0129] Resize: The original collected images disturbed by reflections are uniformly resized, and qualitative cropping is performed if necessary to ensure that all input images have the same dimensions to facilitate subsequent feature separation and feature fusion.

[0130] Instance normalization: The image is instance normalized so that each input sample image has independent normalization parameters, effectively avoiding mutual influence between training samples.

[0131] (2) Separation and extraction of image features

[0132] By using the reflection area detection network and the reflection feature enhancement network based on edge attention, the transmission image is separated from the preprocessed mixed image and its edge features are enhanced. These features can effectively distinguish the reflection elements from the transmission elements in the mixed image, and better reflect the transmission image feature data that needs to be separated.

[0133] (3) Feature fusion generation

[0134] Generative adversarial network construction: Design and implement a generative adversarial network (GAN), which consists of a generative network, a feature converter, and a discriminative network. It can take the pre-extracted input after feature separation and enhancement as input, and learn a better output image without reflection interference.

[0135] Encoder: The main component of the generative network, whose function is to expand and fuse the feature vector of the input projected image features to obtain a generated image with the same scale as the input image after resizing in step S1 and provide it for subsequent module learning and output.

[0136] Feature converter: Converts the image features extracted by the encoder into corresponding feature maps.

[0137] (4) Generate image discrimination

[0138] Decoder: expands the feature vector of the acquired feature map and generates the final output image; as the main component of the discriminator, it participates in the judgment of the quality of the output image and guides the training process of the entire network.

[0139] Through the above detailed image separation, image feature extraction, fusion and reconstruction process, this method can fully capture the differences between the reflected image and the transmitted image in the mixed image, improve the ability to separate and extract the features of the transmitted image, thereby improving the quality and robustness of the entire network structure to remove reflection interference. Especially when this method is used in the certificate acquisition system, it can well adapt to the task of removing reflection interference caused by global and local strong light and polarized light environments.

[0140] Example 1: Dereflection interference in a real border port environment.

[0141] The hardware device using the method of the present invention is operated in a real border inspection port environment to collect images of the licenses and certificates that pass through the pilot in the daily flow. Since the device is in a real port environment, there is a certain amount of external light that affects the imaging quality and is prone to reflection interference, so this embodiment is effective. The entire process of removing reflection interference includes:

[0142] 1. Image preprocessing: Input the image that has been preprocessed with normalized size into the entire network.

[0143] 2. Reflection area detection: used to obtain multi-scale Gaussian-Laplacian features of images.

[0144] 3. Enhancement of reflection features: Output the reflection area probability map and reflection image prediction map by suppressing the reflection-dominated area and enhancing the transmission-dominated area.

[0145] 4. Feature fusion and image generation. The output reflection area probability map and reflection image prediction map enhanced by reflection features are input into the generative adversarial network to obtain the output image after removing reflection interference, and guide the generation of the next round of generators until the output quality meets the requirements.

[0146] Example 2: Dereflection interference in a strong light environment in a laboratory.

[0147] In the laboratory environment, the external environment is changed to strong light, so that the reflection interference is stronger than the general indoor and outdoor environment. The entire process of removing reflection interference is as follows:

[0148] 1. Image preprocessing: Input the image that has been preprocessed with normalized size into the entire network.

[0149] 2. Reflection area detection: The image passes through the reflection area detection module to obtain the multi-scale Gaussian-Laplacian features of the image.

[0150] 3. Reflection feature enhancement. The reflection feature is further refined through the edge attention-based reflection feature enhancement network, and the reflection area probability map and reflection image prediction map are output. Then, the image generation of the next stage is guided by suppressing the reflection-dominated area and enhancing the transmission-dominated area.

[0151] 4. Feature fusion and image generation. The output reflection area probability map and reflection image prediction map enhanced by reflection features are input into the generative adversarial network to obtain the output image after de-reflection interference in a strong reflection light environment. At the same time, the quality of the generated de-reflection interference image is judged to guide the generation of the next round of generators until the output quality meets the requirements.

[0152] These two embodiments demonstrate the wide applicability of the de-reflection interference method based on reflection area detection and edge attention in complex external light source application scenarios. From general indoor and outdoor environments to laboratory environments under strong light source conditions, this method can effectively enhance the system's de-reflection interference capability and improve generalization and robustness.

[0153] like Figure 2 As shown, an embodiment of the present invention provides a de-reflection interference system based on reflection area detection and edge attention to implement the de-reflection interference method based on reflection area detection and edge attention, the system includes an image preprocessing module, a reflection area detection network module, a reflection feature enhancement network module, a generative adversarial network module including a generation network module and a discrimination network module

[0154] The image preprocessing module is used to normalize the size of the image after it is input into the network;

[0155] A reflection area detection network module is connected to the image preprocessing module and is used to obtain multi-scale Gaussian-Laplace features of the image in the reflection area detection network;

[0156] The reflection feature enhancement network module is connected to the reflection area detection network module, and is used to further refine the reflection features through the reflection feature enhancement network that integrates channel attention, and output the probability map of the reflection area and the prediction map of the reflection image to the generative adversarial network;

[0157] The generative adversarial network module includes a generative network module and a discriminative network module, which is used to guide image generation by suppressing the reflection-dominated area and enhancing the transmission-dominated area under the joint action of the generative network and the discriminative network.

[0158] An embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the de-reflection interference method based on reflection area detection and edge attention.

[0159] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method for removing reflection interference based on reflection area detection and edge attention.

[0160] An embodiment of the present invention provides a certificate image data processing terminal, which is used to implement the certificate image acquisition system of the reflection area detection and edge attention based de-reflection interference method.

[0161] The embodiment of the present invention provides a cross-platform computer application program, which can be deployed on Windows, Linux and most domestic operating systems, and is used to complete the steps of the de-reflection interference method based on reflection area detection and edge attention in combination with specific hardware.

[0162] The following are two specific embodiments, showing how to apply the de-reflection interference method based on reflection area detection and edge attention.

[0163] The present invention uses a total of 4 data sets, including 1 real-scene reflective image data set, 2 incremental data sets, and 1 test data set.

[0164] In the model training phase, the present invention uses a synthetic data set, the data sources are SIR containing three real scenarios: 2The data set contains 454 pairs of images and 2,200 pairs of synthetic images in real scenes, including 90 pairs of data sets synthesized by Gaussian blurring of indoor and outdoor scenes, and 200 pairs of Nature data sets shot through portable glass. These data together constitute the training data set named RSRR2944. This part of the data set contains paired transmission images and synthetic images mixed with reflection images, which has a good guiding significance for model training.

[0165] In order to improve the effectiveness of training, this method uses two additional incremental data sets, which can enhance the images during the pre-training process, including one real certificate reflection image data set and one synthetic certificate reflection image data set. The real certificate data set consists of 615 pairs of visible light certificate images with reflection interference collected by the certificate acquisition system mentioned in the present invention and the corresponding directly photographed certificate projection images; the synthetic certificate uses a synthesis method, which specifically regards the captured mixed image I as a linear combination of the transmission image B and the reflection image R, and its mathematical model can be expressed as the following formula:

[0166] I=αB+(1-α)(K*R)

[0167] Where α is a hyperparameter representing the transmittance, and its value is positively correlated with the transmission intensity. The higher the transmission intensity, the closer the acquired image is to the pure transmission image B. K is a Gaussian blur kernel, and * represents a convolution operation. To improve the authenticity of the reflected image, a Gaussian smoothing kernel with a random size from 3 to 15 pixels is used to perform Gaussian blur processing on the reflected image. At the same time, to avoid the problem of image overflow, the sum of the synthesis coefficients of the two is set to 1, and a mixed image containing reflection interference is obtained by linear superposition. The present invention uses commonly used certificate images (712 images) as transmission images and common certificate collection background images (445 images) as reflection images. The certificate image data containing reflection interference is synthesized by the above method and data enhancement processing is performed by flipping, rotating, random cropping, etc., to obtain a total of 5520 pairs of synthetic certificate data sets, and this part of the incremental data set is named PPRR6135.

[0168] In order to effectively evaluate the performance of the algorithm, this method selected a total of 800 images from the domestic and foreign public reflective image test set and the collected license reflective image test data set. 300 images were randomly selected from RSRR2044 as the domestic and foreign public reflective image test set, and 500 images were extracted from the visible light license images collected by our hardware system and PPRR6135 as the collected license reflective image test set. The network structure is implemented using the Pytorch deep learning framework, using the Adam optimizer, and the optimizer parameters β1 and β2 are set to 0.8 and 0.99 respectively. It is trained with a fixed learning rate lr=0.0002, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used to evaluate the image quality. The peak signal-to-noise ratio (unit: dB) is defined based on the mean square error. The image quality is expressed by calculating the mean square error between the input image and the output image, which is written as:

[0169]

[0170] Among them, MAXI represents the maximum pixel value of the image. For RGB images, the MSE of the three channels needs to be calculated separately and then averaged. Generally speaking, the larger the PSNR value, the better the image quality. Structural similarity is a parameter used to measure the similarity between the generated image and the original transmission image. The similarity of the images is measured from three aspects: brightness, contrast, and structure. The formula is:

[0171]

[0172] where μ x and μ y are the pixel averages of the input images x and y, respectively. are the variance of input x and y, σ xy is the covariance of input images x and y, c1 and c2 are constants, so that the range of SSIM is [0,1]. The closer its value is to 1, the higher the similarity of input images x and y. During the test, ablation experiments on reflection area detection and reflection feature enhancement were conducted to illustrate the impact of the module network on the experiment. At the same time, the PSNR of the complete experiment reached 23.11, and the SSIM value reached 0.836, which is ahead of the same type of de-reflection algorithm. Figure 6 exhibit.

[0173] In summary, compared with the existing related methods, the method proposed in the present invention is the first algorithm model that performs reflection area detection and reflection feature enhancement to remove reflection interference. It is superior in accuracy, generated image quality and running time, and is being tested simultaneously on our company's hardware products, which is conducive to the practical application of the algorithm.

[0174] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0175] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for removing reflection interference based on reflection area detection and edge attention, characterized in that: The following steps are involved: S1: The idea of ​​generative adversarial network is adopted in the main body, and the mixed image containing reflection interference is preprocessed as the input of the model; S2: The mixed image containing reflection interference input in S1 is input into the reflection area detection network module, and then passes through the reflection feature enhancement network module to obtain the multi-scale Gaussian-Laplacian features of the image and finally participate in the synthesis of the image gradient features of the enhanced reflection; S3: The image gradient features obtained in S2 are used as the input of the entire generation network, and the features are extracted through the shallow feature extraction network; S4: The features extracted in S3 are transmitted to the discriminant network via skip connections. While obtaining the output image after de-reflection interference, the quality of the generated image after de-reflection interference is judged to guide the generation of the next round of generators until the output quality meets the requirements.

2. The method for removing reflection interference based on reflection area detection and edge attention as claimed in claim 1, characterized in that: The S1 specifically includes: The task of removing image reflection interference usually considers that the image containing reflection interference is a mixture of the transmission image and the reflection image. When the traditional method of calculating the mixture ratio of the two through some image features or using a mixing matrix to separate them is used, it has great limitations in processing the task of removing reflection interference from the license image, which is specifically characterized by the inability to process images with uneven reflection intensity and poor generalization ability. Therefore, an image reflection interference removal algorithm based on deep learning is used to achieve this. Since the generative adversarial network can effectively enhance image details in the generation stage and can also guide the image generation effect by adjusting the attention distribution, improvements are made based on the generative adversarial network. The mixed images with reflection interference collected in the license acquisition hardware system are resized and instance normalized to ensure that the images input to the network are of the same size and have no influence on each other; instance normalization refers to the process of calculating the mean and standard deviation of each input feature map on a single channel, and then using the mean and standard deviation on each channel to normalize the elements in the single channel; instance normalization has great advantages in image style transfer. For a feature map x with a batch size of N, a length, width, and number of channels of h, w, and c, respectively, its specific mathematical definition is as follows: where x hw represents an image of scale h×w under a single channel, μ(x) and σ(x) are the mean and variance respectively, γ and β are variables learned during training, and ε is a small constant.

3. The method for removing reflection interference based on reflection area detection and edge attention as claimed in claim 1, characterized in that: The S2 specifically includes: According to the characteristic that the gradient of the reflected image in the mixed image is sparser than that of the transmitted image, a reflection area detection network is designed. The mixed image passes through the multi-scale reflection area detection network to obtain the multi-scale Gaussian-Laplacian features of the image, and then further refines the reflection features through the reflection feature enhancement network that integrates channel attention, outputs the reflection area probability map and the reflection image prediction map, and then guides the image generation of the next stage by suppressing the reflection dominant area and enhancing the transmission dominant area; According to the phenomenon that the edge features of the projected image are usually more obvious than those of the reflected image, a reflection feature enhancement network based on edge attention is designed to enhance the edge of the transmitted image; the reflection feature enhancement module has two parts of input, one of which is the multi-scale Gaussian-Laplacian feature I of the original image obtained in the reflection area detection module out , and the second is the output I of the image completely passing through the reflection area detection module RD , I out The feature enhancement is performed through four cascaded squeeze-excitation residual modules and then combined with I RD The weighted product obtains the image gradient feature of the enhanced reflection; Removing reflection interference can be regarded as the task of separating the reflection image and the transmission image in the mixed image. In the traditional algorithm of removing interference by using image transformation, the selected prior conditions are highly dependent on the scene. The prior conditions may fail in a specific scene, which directly affects the effect of reflection removal. In order to ensure the acquisition of multi-scale image gradient features, a multi-scale feature extraction module is used in the network structure. The input image is scaled by multi-scale global pooling, and the image size is scaled to 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original scale. The original scale image and the multi-scale image obtained by the pooling operation are respectively subjected to Gaussian-Laplacian convolution to obtain multi-scale image gradient features. Laplace convolution is a common edge detection method. It obtains the edge features of the image by performing second-order differential on the image, but it is sensitive to image noise and discrete points. In order to overcome this problem, Gaussian smoothing is usually performed before Laplace convolution. The convolution kernel is calculated as follows: where σ 2 is the standard deviation, and its selection is positively correlated with the image smoothing effect. 2 Fine-tune the value; The obtained multi-scale Gaussian-Laplacian feature maps are upsampled to the original scale size and spliced ​​using the bilinear interpolation algorithm; To represent the image of the corresponding scale after the downsampling operation, use Represents the upsampled image of the corresponding scale. The output image is: Use 3 squeeze-excitation residual modules to enhance the output image I out The multi-scale Gaussian-Laplacian features of the image are obtained by the squeeze-excitation residual module through the two steps of squeeze and excitation to model the interdependence between the channels of the image. The squeeze is performed by performing global average pooling on the feature map to obtain the global compressed feature vector of the current feature map. For the input image X∈H×W×C, the squeeze step can be written as the following formula: Where Z c refers to the output related to the cth channel, input x c From the convolutional layer; the excitation is to obtain the weight of each channel in the feature map through two fully connected layers, activate it through the Sigmoid function, and then use the weighted feature map as the input of the next layer of the network. The excitation step can be written as the following formula: T1 and T2 are two linear layers, and finally the feature strength of each channel is adaptively adjusted through the global loss function of the network; After obtaining the enhanced image features, the image features are activated by the PReLU function. The PReLU function formula can be written as: where y i is the input of the ith channel, a i Control the slope of the negative semi-axis, the initial value is a i = 0.2, its value can be adaptively updated through back propagation in the subsequent training process, and the process of feature enhancement and activation is recorded as f LOG Finally, through the 3×3 convolution kernel, the Sigmoid activation normalization is used to obtain the final reflection area detection probability map I RD , I RD can be expressed as: I RD =Sigmoid(Conv(f LOG (I out ))) Thus, as shown in FIG3 , the mixed image can obtain preliminary reflection image gradient features through the reflection area detection module, which provides basic clues for predicting the reflection image. In order to better predict the reflection image, the reflection feature enhancement module is used to further refine the gradient features of the reflection image. The reflection feature enhancement module group has two parts of input, one of which is the original image obtained by the multi-scale Gaussian-Laplacian feature extraction module. out , and the second is the final output I of the reflection area detection module RD , feature enhancement is performed through four cascaded squeeze-excitation residual modules, and then combined with I RD The weighted product is used to further enhance the gradient features of the reflected image; its specific network structure can be seen in Figure 4. The reflection feature enhancement module plays a role in suppressing the transmission image features, which is helpful for the subsequent prediction of the reflected image. In order to enhance the edge representation ability of the network, an edge attention module is introduced into the network. In traditional edge detection schemes, only the binary contour of the image is usually extracted. However, in the task of removing reflection interference, obtaining the binary edge contour is not effective for the subsequent restoration of the transmission image, because in the strong reflection area, there may be indistinguishable hard boundaries that interfere with the edge detection of the actual transmission image. Therefore, the average absolute color difference method is chosen to obtain the image edge representation. The color difference between a certain pixel and the surrounding pixels is calculated in the three RGB channels of the input image and the weighted average is calculated. The image edge is estimated based on the average absolute color difference. The specific calculation method is as follows: Where (x, y) is the coordinate of the corresponding pixel in the image, and c refers to the RGB color space channel of the image. The edge map of the mixed image obtained in this way is more effective in suppressing the hard boundaries of the reflective area than the binary edge map obtained by graying the image, because the color gamut of the hard boundaries of the reflective area is usually relatively single and has little impact on the mean absolute color difference. The obtained edge map is introduced into the network as an additional channel input to increase the edge attention of the network.

4. The method for removing reflection interference based on reflection area detection and edge attention as claimed in claim 1, characterized in that: The S3 specifically includes: The generative network consists of a convolutional layer, a ReLU activation layer, an instance normalization layer, and a coordinated attention layer. The generative network extracts features from the input image. To enhance the image feature extraction capability, a coordinated attention module is introduced into the generative network. This module aggregates the features in the vertical and horizontal directions into two separate feature maps by performing pooling operations in the row and column dimensions. The acquired feature maps are encoded into two attention maps, each of which can capture long-distance dependencies in the row or column spatial direction. This attention mechanism can efficiently integrate the global features of the image, not only to obtain cross-channel feature information, but also to obtain coordinate-sensitive information, which helps the model identify and locate features of interest. In the feature extraction link of the mixed image, the coordinate positioning of the reflection area is an important basis for removing reflection interference. Channel attention solves the problem of feature acquisition, and spatial attention solves the problem of feature positioning. Coordinated attention provides the network with both spatial attention and channel attention, while only increasing a small amount of computational complexity, so that the generative network can efficiently obtain the features of the transmission image. After obtaining the separated and enhanced transmission image features through S2, it is used as the input of the generative adversarial network for subsequent feature reconstruction and output; the generative adversarial network is a generative model that combines the characteristics of the generative model and the discriminative model. The network consists of a generator network and a discriminator network. The generator is responsible for generating new data samples according to the characteristic distribution of the sample data, and the discriminator gives feedback to further optimize the generator. The two neural networks learn by playing against each other, and finally guide the data samples generated by the generator to be as close to the real data samples as possible; the network can be used in an unsupervised or semi-supervised manner, which greatly reduces the requirements for sample data; The basic process of generating adversarial networks is that the generator network G learns the random input z of the model, the generator outputs the generated sample G(z), and then inputs the generated sample and the real sample G(x) into the discriminator. The discriminator marks the real sample as 1 and the generated sample as 0. The purpose is to train a maximum discriminator D to classify the sample accurately, and on this basis, minimize the adversarial loss log(1-D(G(z))) to optimize the generator G, thereby reducing the accuracy of the discriminator classification. The valuation functions of the two are completely opposite, and a zero-sum game is performed according to the minimization and maximization valuation function V(G,D), which can be written as: Where x represents the real sample data, p data (x) represents the true sample data distribution, z represents random input data, p z (z) represents the prior distribution of random input data; when p z =p data When , there is an optimal solution to the zero-sum game.

5. The method for removing reflection interference based on reflection area detection and edge attention as claimed in claim 1, characterized in that: The S4 specifically includes: The main function of the discriminant network is to expand the feature vector of the acquired feature map, and finally obtain a generated image with the same scale as the input image; in addition, jump connections are introduced in the generation network and the discriminant network, which can directly transfer image data from the generation network to the discriminant network to reduce the loss of image detail information; in addition, a feature converter composed of 6 cascaded residual modules is added between the generation network and the discriminant network. Its main function is to convert the image features extracted by the discriminant network into the corresponding feature map, and it can effectively alleviate the problem of gradient disappearance during training; The decoder consists of a deconvolution layer, a ReLU activation layer, a normalization layer, a Tanh activation layer, etc. The main function of the decoder is to expand the feature vectors obtained from the feature converter and the jump connection to obtain a generated image with the same scale as the input image. At the same time, it participates in the discrimination process of the entire generative adversarial network, and improves the refinement and generation quality of the entire network through the adversarial learning process.

6. A system for removing reflection interference based on reflection area detection and edge attention, which implements the method for removing reflection interference based on reflection area detection and edge attention as claimed in any one of claims 1 to 5, characterized in that: The system includes an image preprocessing module, a reflection area detection network module, a reflection feature enhancement network module, a generative adversarial network module including a generation network module and a discrimination network module. The image preprocessing module is used to normalize the size of the image after it is input into the network; A reflection area detection network module is connected to the image preprocessing module and is used to obtain multi-scale Gaussian-Laplace features of the image in the reflection area detection network; The reflection feature enhancement network module is connected to the reflection area detection network module, and is used to further refine the reflection features through the reflection feature enhancement network that integrates channel attention, and output the probability map of the reflection area and the prediction map of the reflection image to the generative adversarial network; The generative adversarial network module includes a generative network module and a discriminative network module, which is used to guide image generation by suppressing the reflection-dominated area and enhancing the transmission-dominated area under the joint action of the generative network and the discriminative network.

7. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the de-reflection interference method based on reflection area detection and edge attention as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method for removing reflection interference based on reflection area detection and edge attention as described in any one of claims 1 to 6.

9. A certificate image data processing terminal, the terminal being used to implement the certificate image acquisition system of the de-reflection interference method based on reflective area detection and edge attention as described in claim 6.

10. A cross-platform computer application that can be deployed on Windows, Linux and most domestic operating systems, and is used in combination with specific hardware to complete the steps of the de-reflection interference method based on reflection area detection and edge attention as described in any one of claims 1 to 6.

Images