A scanning transmission image restoration method, device and storage medium

Abstract

The present invention discloses a scanning transmission image restoration method, device and storage medium, which relates to the field of image processing. The present invention mainly utilizes the cycle consistency generative adversarial network in the field of deep learning to perform scanning image restoration, considers the complexity of text distribution in actual scanning transmission images, combines human visual characteristics, integrates channel attention mechanism and regional attention mechanism into the network, non-uniformly processes different features and different regions, utilizes the constraint transfer learning and cyclic structure of the cycle consistency generative adversarial network, and finally completes the unsupervised scanning transmission image restoration task of unpaired images. The present invention has good image restoration performance and has good robustness and adaptability to scanning transmission image data in various situations.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a scanning transmission image restoration method, device and storage medium. Background Art

[0002] Scanners are the main way to solve the problem of digitizing various documents, photos, slides, negatives and other manuscripts since the 1980s. They are widely used in light printing, newspapers, publishing houses, document production, graphic database management and other work. With the development of the Internet and multimedia and the higher standards for the image quality of digital documents, for overlapping scanning using the transmission scanning principle, the content on the back is usually transmitted to the content on the front, thus affecting normal reading and use.

[0003] At present, the problems of scanning transmission image restoration are:

[0004] 1. The traditional method is to restore the image by detecting the peak value of the grayscale histogram of the scanned image. The threshold setting is not unique, which makes the restoration process difficult to achieve and the result is not ideal;

[0005] 2. The image restoration method based on the generative adversarial network can restore the scanned transmission image without using a threshold, and evenly process the scanned transmission image so that the area without transmission is over-processed, resulting in color distortion;

[0006] 3. The training-based learning method requires a large number of paired data sets during the training process. The data sets are difficult to collect and there is overfitting and difficulty in generalization.

[0007] Based on the above problems, it is still difficult for existing scanners on the market to effectively remove the scanned transmission image information. Summary of the invention

[0008] The present invention provides a scanning transmission image restoration method, device and storage medium. The present invention uses a cycle-consistent generative adversarial network with an attention mechanism to "non-uniformly" process the scanning transmission image, and does not require the use of paired data sets during network training. See the following description for details:

[0009] In a first aspect, a scanning transmission image restoration method is provided, the method comprising:

[0010] The A-CycleGAN network model consisting of the first and second generators and the first and second discriminators is trained using a synthetic transmission image training set;

[0011] A first non-transmission image is generated and outputted by the first generator. If the first discriminator cannot recognize all the first non-transmission images, a first error is calculated by adversarial loss, and the parameters of the first discriminator are updated by back propagation to seek a first minimized classification error.

[0012] The first discriminator is fixed, and the first transmission image is generated and outputted through the second generator. The difference between the input synthetic transmission image and the first transmission image is calculated through the cycle consistency loss and the perceptual loss, and the first loss value is obtained. The network parameters of the first generator are updated through back propagation to seek the first global optimal value.

[0013] The second generator generates and outputs a second transmission image. If the second discriminator cannot recognize all the second transmission images, a second error is calculated by adversarial loss, and the parameters of the second discriminator are updated by back propagation to seek a second minimized classification error.

[0014] The second discriminator is fixed, and a second non-transmission image is generated and outputted through the first generator. The difference between the input non-transmission image and the second non-transmission image is calculated through the cycle consistency loss and the perceptual loss, and the second loss value is obtained. The network parameters of the second generator are updated through back propagation to seek the second global optimal value.

[0015] When all training is completed, the generator network outputs the final non-transmission image.

[0016] Wherein, the A-CycleGAN network model includes:

[0017] Transmission-transmission branch: Use the input synthetic transmission image h to generate the first non-transmission image d through the first generator h , reconstructed into the first transmission image h′ through the second generator;

[0018] No-transmission-no-transmission branch: Use the input no-transmission image d to generate the second transmission image h through the second generator d , reconstructed into a second non-transmission image d′ through the first generator;

[0019] The two branches share the first and second generators, the transmission-transmission branch uses the first discriminator, and the no-transmission-no-transmission branch uses the second discriminator.

[0020] Furthermore, the first generator includes: a channel attention mechanism block and a regional attention mechanism block, and uses a jump connection structure to enable the high layer to obtain the image feature information of the low layer.

[0021] Furthermore, the first generator is built using convolution, dense blocks and attention mechanism structures. The dense blocks are used to extract all features from shallow to deep layers. The dense blocks are composed of three groups of structural units, and each group of structural units includes: structural blocks and attention mechanism networks.

[0022] The method uses unpaired images to perform scanning transmission image restoration tasks during network training.

[0023] In a second aspect, a scanning transmission image restoration device is provided, the device comprising: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0024] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the method steps described in the first aspect.

[0025] The beneficial effects of the technical solution provided by the present invention are:

[0026] 1) Compared with the traditional method, the scanning transmission image restoration method provided by the present invention uses unpaired images to perform the scanning transmission image restoration task during the network training process, thus solving the problem of difficulty in data set collection;

[0027] 2) Incorporating channel attention mechanism and region attention mechanism into the generative adversarial network can improve the representation of the focus point and the expression of the content of interest, so as to achieve better separation effect of scanned transmission images, reconstruct detailed content and ensure low color distortion;

[0028] 3) The scanning transmission image restoration method provided by the present invention has high robustness and stability. It not only has significant restoration performance for transmission images from a single scanner source, but also has high generalization ability for large-scale scanning transmission image data sets from multiple scanner sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of a scanning transmission image restoration method;

[0030] Figure 2 Schematic diagram of the overall structure of the adversarial network for improved cycle consistency generation;

[0031] Figure 3 Generate the generator G in adversarial networks for improved cycle consistency d Schematic diagram of the structure;

[0032] Figure 3(a) is a schematic diagram of the structure of the channel attention mechanism block;

[0033] Figure 3 (b) is a schematic diagram of the structure of the regional attention mechanism block;

[0034] Figure 4 Generate the generator G in adversarial networks for improved cycle consistency h Schematic diagram of the structure;

[0035] Figure 5 Schematic diagram of the structure of the discriminator in the adversarial network for improved cycle consistency generation;

[0036] Figure 6 To artificially synthesize scanning transmission images and restoration effect images;

[0037] Among them, (a) is a schematic diagram of an artificially synthesized scanned transmission image; (b) is a diagram of the restoration effect of the artificially synthesized scanned transmission image using the A-CycleGAN method.

[0038] Figure 7 To artificially synthesize scanning transmission images and restoration effect images;

[0039] Among them, (a) is a schematic diagram of an artificially synthesized scanning transmission image; (b) is a restoration effect diagram of the artificially synthesized scanning transmission image using the A-CycleGAN method; (c) is a restoration effect diagram of the artificially synthesized scanning transmission image using the traditional histogram threshold method.

[0040] Figure 8 To scan the transmission image and restore the effect diagram;

[0041] Among them, (a) is a schematic diagram of the scanned transmission image obtained by the FS-1020MFP scanner; (b) is a restoration effect diagram of the scanned transmission image obtained by the FS-1020MFP scanner using the A-CycleGAN method; (c) is a restoration effect diagram of the scanned transmission image obtained by the FS-1020MFP scanner using the traditional histogram threshold method.

[0042] Fig. 9 It is a schematic diagram of the structure of the scanning transmission image restoration device. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0044] The following first explains and illustrates the technical terms involved in the embodiments of the present invention:

[0045] Generator: Used to learn the distribution of real images so that the images generated by itself are more realistic and can "fool" the discriminator.

[0046] Discriminator: It is necessary to distinguish whether the received images are real or fake.

[0047] Nash equilibrium: The images generated by the generator are close to the real image distribution, while the discriminator cannot distinguish between real and fake images, and the probability of predicting a given image as real is close to 50%.

[0048] Neuron: In artificial neural networks, a neuron is a processing unit in the network, a placeholder for a mathematical function that applies a function to an input and produces an output.

[0049] Convolution layer: The convolution calculation formula is: N = (W-F + 2P) / S + 1, where N represents the number of output channels, W represents the number of input channels, F represents the size of the convolution kernel, P represents the size of the padding value, and S represents the step size. The convolution kernel is a matrix with a fixed weight of F × F pixels. It moves horizontally and vertically in the pixel matrix of the original image according to the preset step size (i.e. from left to right, from top to bottom), and performs element-by-element dot multiplication of the data of different data windows in the original image and the convolution kernel, and then adds them together to perform a convolution operation. The padding value is introduced to fill the pixels around the original image (the pixel value of the pixel is preset to 0) to avoid the problem of image shrinkage and loss of image edge information after the convolution operation.

[0050] Pooling layer: used to reduce image size and increase computing speed. It takes the maximum value or average value within the filter sliding area.

[0051] ReLU activation layer: The expression is: f(x) = max(0,x), where x represents the input value of this layer of the network. The ReLU function is a piecewise linear function that changes all negative values ​​to 0, while the positive values ​​remain unchanged. This operation is called unilateral inhibition, which activates some neurons in the neural network to improve computing efficiency.

[0052] LeakyReLU activation layer: The expression is: f(x) = max(a*x, x), where a represents a fixed parameter in the interval (0, 1). In the embodiment of the present invention, a = 0.2 is taken. It is used to solve the problem that the neuron does not learn after the ReLU function enters the negative interval.

[0053] Sigmoid activation layer: The expression is: When x approaches negative infinity, f(x) approaches 0; when x approaches positive infinity, f(x) approaches 1; when x=0, f(x)=0.5.

[0054] Normalization layer: Normalization is to scale the value of a column of numerical features in the training set to between 0 and 1. After data normalization, the optimization process of the optimal solution will obviously become smoother, and it is easier to correctly converge to the optimal solution.

[0055] Attention mechanism: Human vision quickly scans the global image to obtain the focus of attention on the target area that needs to be focused on, and then invests more attention resources in this area to obtain more detailed information about the target that needs to be focused on, while ignoring other useless information, which greatly improves the efficiency and accuracy of visual information processing. Incorporating the attention mechanism on the basis of the generative adversarial network can help the model assign different weights to each part of the input, extract more critical and important information, and enable the model to make accurate judgments without incurring large overhead in the calculation and storage of the model.

[0056] Example 1

[0057] In order to solve the problem that the text content on the back of thin paper printed or written materials (such as ordinary paper books, textbooks, and test papers) may be transmitted to the front page during the scanning process, thus affecting the normal reading of readers, the embodiment of the present invention provides a scanning transmission image restoration method, see Figure 1 , the method comprises the following steps:

[0058] 101: Build a preset network consisting of a generator network and a discriminator network;

[0059] The preset network includes: a generator network and a discriminator network, and the generator network includes: a first generator G d and the second generator G h ; The discriminator network includes: the first discriminator D d and the second discriminator D h , the generator network and the discriminator network are set to adversarial learning mode. Both the generator network and the discriminator network are deep neural networks. The first generator G d is a deep neural network with skip connections and attention mechanism; the second generator G h It is a well-known deep neural network with convolution and deconvolution operations, that is, a convolutional neural network is used to extract features from an input image, and then a deconvolution layer is used to restore low-level features from a feature vector.

[0060] 102: Calculate the loss function value;

[0061] Among them, the real image without transmission is combined with the first generator G d The output non-transmitted image is used as the first image pair, the real image containing the transmission and the second generator G h The generated transmission image is used as the second image pair, and the first and second image pairs are input into the first discriminator D d and the second discriminator D hThe false discrimination probability and the true discrimination probability are calculated through the discriminator network, and the loss function formula of the discriminator network and the loss function formula of the generator network are used to obtain the loss function value of the discriminator network and the loss function value of the generator network respectively.

[0062] 103: Update network parameters;

[0063] The loss function values ​​of the discriminator network and the generator network are minimized respectively, and the deep neural network back-propagation algorithm is used to iteratively update the values ​​of the parameters of the generator network and the discriminator network until the termination conditions are met, thus completing the training of the generator network and the discriminator network.

[0064] Among them, the deep neural network backpropagation algorithm is well known to those skilled in the art and will not be elaborated in detail in the embodiments of the present invention.

[0065] 104: Generate a transmission image restoration map of the test set;

[0066] When the generator network training is completed, the transmission image is input into the trained generator network. Through iterative calculation in the generator network, the generator network outputs the restored image of the transmission image of the test set.

[0067] 105: Model training: Use the synthetic transmission image training set to train the A-CycleGAN network model;

[0068] Among them, the A-CycleGAN network model is mainly divided into two parts, one is the generator network and the other is the discriminator network. The function of the generator network is to generate the image specified by the specified domain according to the input original domain image and the label information of the specified domain. The function of the discriminator network is to classify the input image, determine whether it is a real image or a generated image, and output the domain information to which it belongs.

[0069] 106: Input the transmission image h, the transmission image h passes through the first generator G d Generate and output the non-transmission image d h , in the second discriminator D d Determine the generated image d h Is it a non-transmission image? If the second discriminator D d Not all generated images d h Identify it as a non-transmission image, calculate the error using the following adversarial loss formula (1), and back-propagate to update the second discriminator D d parameters, seeking to minimize the classification error.

[0070]

[0071] in, represents the expected value of the distribution function, h and d represent the samples in the transmission image dataset and the non-transmission image dataset, respectively, and the two are not required to be paired, h~p data (h) represents the probability distribution of samples in the transmission data set, d~p data (d) represents the probability distribution of samples in the no-transmission dataset, G d and G h represents the generator, D d and D h represents the discriminator.

[0072] 107: Fixed D d Parameters, no transmission image d h Through the second generator G h Generate h', calculate the difference between h' and h through the cycle consistency loss formula (2) and the perceptual loss formula (3), calculate the loss value, and back propagate to update the first generator G d The network parameters are used to find the global optimal value;

[0073] Among them, the first generator G d Including: channel attention mechanism block and regional attention mechanism block. In order to generate clear and realistic images and obtain better results in details and edges, the use of jump connection structure allows the high-level layer to obtain the image feature information of the underlying layer, so that the high-level layer has more image detail information, thereby restoring a better clean image.

[0074]

[0075] Among them, ||*|| 1 is 1 paradigm,

[0076]

[0077] Among them, ||*|| 2 For 2 normal form, Represents the feature extractor of the VGG-16 network. The perceptual loss function combines the features extracted from the 2nd and 5th pooling layers of the VGG-16 network, which is initialized by the ImageNet pre-trained model.

[0078] 108: Incorporate channel attention mechanism blocks and regional attention mechanism blocks to achieve better scanning transmission image separation effect;

[0079] In order to further realize the above invention, in the first generator G dIn the network, in order to achieve better restoration results in details and colors, channel attention mechanism blocks and regional attention mechanism blocks are integrated to achieve better scanning transmission image separation effects. At the image region level, the network needs to be able to capture the high response areas in the feature map, so that the model pays more attention to the corresponding areas of the feature map and processes this feature area. The regional attention mechanism can realize the response of information in different regions.

[0080] Among them, the channel attention mechanism focuses on different features of the image. Integrating the attention mechanism into the network can better reconstruct the details and ensure low color distortion. Using skip connections avoids directly calculating the loss in the high-level feature map, and combines the features in the low-level feature map, so that the final feature map contains both high-level features and many low-level features, realizing the fusion of features at different scales and improving the accuracy of the model.

[0081] 109: In the channel attention mechanism module, the feature map size is converted from C×H×W to C×1×1, where C represents the number of channels and H×W represents the image size;

[0082] Features are obtained through two convolutional layers and ReLU and Sigmoid activation functions. The processing process of the channel attention mechanism block is shown in formula (4):

[0083]

[0084] Where pool(·) represents the pooling layer, Conv represents the convolutional layer, σ is the Sigmoid function, δ is the ReLU function, and F is the input of the channel attention mechanism block. is element-wise multiplication, F c Represents the output of the channel attention mechanism. The regional attention mechanism module converts the image of shape C×H×W into 1×H×W, reducing the number of channels in the original feature to 1. In this way, the network pays different attention to each pixel in the region, thereby achieving the response of different regional information. The processing process of the regional attention mechanism block is shown in formula (5):

[0085]

[0086] Among them, F result Represents the output of the region attention mechanism block.

[0087] 110: Input the non-transmission image d, and the non-transmission image d passes through the second generator G h Generate and output transmission image h d , in the second discriminator D h Determine the generated h d Is it a transmission image? If the second discriminator D hNot all generated h d Identify it as a transmission image, calculate the error through the adversarial loss formula (6), and back propagate to update the discriminator D h parameters, seeking to minimize the classification error.

[0088]

[0089] Fixed D h Parameter, h d Through the generator G d Generate d′, calculate the difference between d′ and d through the cycle consistency loss formula (2) and the perceptual loss formula (3), obtain the loss value, and back-propagate to update the generator G h Network parameters, seeking global optimal values.

[0090] 111: Steps 101 and 102 and steps 103 and 104 are simultaneously subjected to cyclic iterative training, and the training is stopped when both the generator network and the discriminator network reach a Nash equilibrium state;

[0091] 112: Generate test set transmission image restoration map: When the network G is generated d and G h After the training is completed, the real image containing transmission and the real image without transmission to be restored are received and input into the trained generator network. Through iterative calculation in the network, the generator network G d Output the transmission image restoration map of the test set.

[0092] In summary, the embodiments of the present invention utilize a cycle-consistent generative adversarial network that incorporates an attention mechanism to "non-uniformly" process scanned transmission images, and do not require the use of paired data sets during network training.

[0093] Example 2

[0094] The scheme in Example 1 is further introduced below in combination with specific calculation formulas, drawings, and examples. See the following description for details:

[0095] 201: The scanning transmission image restoration problem can be regarded as a conversion problem from image to image, that is, the problem of converting a scanning transmission image into an unscanned transmission image;

[0096] In the process of training the model, a large number of scanning transmission image datasets are required as the data basis. However, there is no scanning transmission image dataset that can be used directly in reality. Therefore, the first step is to build a scanning transmission image dataset. In order to ensure the good generalization performance of the subsequent model, the types and styles of scanning transmission images should be as many as possible. Text materials of different styles are collected to make scanning transmission images. In order to ensure the generality of the dataset, the official dataset publicly available on the Internet is selected as the original non-scanning transmission image. A synthetic scanning transmission image is obtained by mixing the two images, in which part of the image is horizontally flipped 90° as the back image of the transmission. The mixing ratio of the front image and the back image is randomly selected in the range of 0.15 to 0.25 during the mixing process to simulate the different transmission of each sheet of paper in the printer in real life, thereby obtaining a synthetic scanning transmission image dataset.

[0097] 202: Using synthetic transmission dataset to train A-CycleGAN network model;

[0098] like Figure 2 The A-CycleGAN network model shown in the figure has two main branches: (1) Transmission-Transmission Branch: h→d h →h′, use the transmitted image h to generate the non-transmitted image d h , reconstructing the transmission image h′. (2) No transmission-no transmission branch: d→h d →d′, use the non-transmitted image d to generate the transmitted image h d , and then reconstructed into a non-transmitted image d′. Each of the two branches of A-CycleGAN has a discriminator D h and D d , these two branches share two generators G d and G h . G d The transmission image is restored to the non-transmission image, and G h The transmission part is added to the non-transmission image. h and D d network, and the generator G h The network is a commonly used structure in deep learning. Generator G d Dense blocks and attention mechanisms are added to the network model structure to ensure that the restoration method has good image restoration performance.

[0099] 203: Figure 3 is the generator G used in this embodiment d Network model structure diagram. Figure 3 As shown, in this embodiment, the generator G dThe network is mainly built using convolution, dense blocks and channel attention mechanism structures, among which the convolution structure is a commonly used structure in deep learning. Dense blocks are mainly used to extract all features from shallow to deep layers, and the attention mechanism focuses on effective information such as areas with heavy background transmission content and high-frequency textures, thereby retaining more detailed information of the image and restoring a clear image. After dense block processing, the output is a 3×64×64 pixel image, which is then input into the attention mechanism network.

[0100] 204: The dense block structure consists of three groups of structural units, each of which includes: a structural block (i.e., including: the first convolutional layer, the activation layer and the second convolutional layer), an attention mechanism network, Figure 3 The middle dense block structure is a structural diagram of the dense block in the embodiment of the present invention. In order to achieve a good balance between performance and running time, the number of channels of all intermediate convolutional layers is set to 64, the convolution kernel size is set to 3×3 pixels, the convolution kernel movement stride is 1, and the feature map boundary padding is 1. When the image feature information passes through all layers of the network, a large amount of unnecessary redundant feature information will be exchanged, resulting in the loss of some important feature information. Therefore, the embodiment of the present invention designs a jump connection in the dense block to fuse the shallow features with the deep features to help the network retain more detailed information of the image, thereby restoring a clear image.

[0101] 205: The attention mechanism network in step 204 includes: a channel attention mechanism block and a regional attention mechanism block. The channel attention mechanism block and the regional attention mechanism block are integrated to achieve a better scanning transmission image separation effect. At the image region level, the network needs to be able to capture the high response area in the feature map, so that the model pays more attention to the corresponding area of ​​the feature map and processes this feature area. Using jump connections, the fusion of features at different scales is achieved, which improves the accuracy of the model results.

[0102] 206: Figure 3 (a) In the channel attention mechanism block, the feature map size is converted from C×H×W to C×1×1, where C represents the number of channels and H×W represents the size of each channel feature map. Features are obtained through two convolutional layers and ReLU and Sigmoid activation functions. The processing process of the channel attention mechanism block is shown in the above formula (4).

[0103] 207: Figure 3 (b) In the regional attention mechanism block, the image with a shape of C×H×W is converted to 1×H×W, so that the number of channels in the original feature is reduced to 1. In this way, the network pays different attention to each pixel in the region, thereby achieving response to information in different regions. The processing process of the regional attention mechanism block is shown in the above formula (5).

[0104] 208: Figure 4is the generator G used in this embodiment h Network model structure diagram. Figure 4 As shown, G h The network model structure adopts an encoder-decoder network. Three convolution blocks are used in the encoding process, and three deconvolution blocks are used in the decoding process. In the encoder network, the number of channels in the first layer is 32, the convolution kernel size is 7×7 pixels, and the convolution kernel movement stride is 1. The number of channels in the second layer is 64, the convolution kernel size is 3×3 pixels, and the convolution kernel movement stride is 2. The number of channels in the third layer is 128, the convolution kernel size is 3×3 pixels, and the convolution kernel movement stride is 2. The convolution kernel size of each layer of the decoder network is 3×3 pixels, the number of channels is 64 and 32 respectively, the convolution kernel movement stride is 1 / 2, and the number of residual blocks is 6.

[0105] 209: Figure 5 is a diagram of the discriminator network model structure used in this embodiment. Figure 5 As shown in the figure, the discriminator network of the A-CycleGAN network model is mainly composed of a convolutional structure, and the input is the output of the generator network, that is, a 3-channel color image of size 256×256 pixels. The discriminator network has five convolutional layers connected in sequence. The first four convolutional layers include convolution, LeakyReLU activation function and normalization layer. The convolution kernel size is 4×4 pixels, the convolution kernel movement stride is 2, the feature map boundary padding is 1, and the number of channels is 64, 128, 256, and 512 respectively. The convolution kernel size of the last convolution layer is 4×4 pixels, and the convolution kernel movement stride is 1, which is used to determine the authenticity of the image.

[0106] 210: In the training of the A-CycleGAN network model in this embodiment, the size of the training set images is uniformly adjusted to 256×256 pixels, the network training is iterated 50 times, and the network training optimization adopts the Adam optimization algorithm, where the learning rate is 0.0001.

[0107] 211: Construct an improved cycle consistency generative adversarial network and train the consistent generative adversarial network; the target loss function of the network is shown in formula (7):

[0108]

[0109] in, represents the target loss function of the network, represents the cycle consistency loss function, represents the perceptual loss function, ω represents the weight coefficient of the perceptual loss function, represents the loss function of the first generative adversarial network, Represents the loss function of the second generative adversarial network. In the task of scanning transmission image restoration, it is necessary to minimize the loss function, the cycle consistency loss function to ensure that the separated image can be restored to the scanning transmission image. The cycle consistency loss function is expressed as the above formula (7).

[0110] In the embodiment of the present invention, there are two discriminators D d and D h , where D d is to distinguish the input non-transmission image from the generated non-transmission image, and D h It is to distinguish the input image with transmission and the generated image with transmission, D d The adversarial loss function is defined as shown in the above formula (1).

[0111] Among them, the discriminator D d The role of is to maximize the loss and distinguish the generated non-transmission image from the real non-transmission image. d The role of is to minimize the loss so that the generated non-transmission image is close to the real non-transmission image. h The adversarial loss function is defined as shown in the above formula (6).

[0112] Since there is a lot of loss in the content of the scanned transmission image, the cycle consistency loss function in the cyclic adversarial generative network is difficult to recover all the texture information. By introducing the perceptual loss function, the image to be generated can be more semantically similar to the target image. The perceptual loss function is expressed as shown in the above formula (3).

[0113] The perceptual loss function combines the features extracted from the 2nd and 5th pooling layers of the VGG-16 network, which is initialized by the model pre-trained on ImageNet. After weighing the pros and cons, the perceptual loss function and the cycle consistency loss function are integrated into the scanned transmission image restoration model based on the generative adversarial network loss function, so that the restored scanned transmission image can be closer to the original image in terms of details, texture and color.

[0114] 212: Model testing: After completing the training of the A-CycleGAN network model, the scanned transmission image dataset is input into the trained network for testing and the restoration results are output.

[0115] In summary, the embodiment of the present invention uses unpaired images to perform the scanning transmission image restoration task during the network training process through steps 201 to 212, thereby solving the problem of difficult data set acquisition; incorporating the channel attention mechanism and the regional attention mechanism into the generative adversarial network can improve the representation of the focus point and the expressiveness of the content of interest, so as to achieve better scanning transmission image separation effect, reconstruct detailed content and ensure low color distortion.

[0116] Example 3

[0117] The feasibility of the schemes in Examples 1 and 2 is verified in combination with specific evaluation criteria, drawings and tables, as described below:

[0118] In order to objectively evaluate the restoration effect of the transmission image, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as the evaluation criteria for the restored image quality. Given an image I and an image K of size m×n pixels, PSNR and SSIM are defined as follows:

[0119] (1)

[0120] in, is the mean square error, I(i, j) and K(i, j) are the pixel values ​​of image I and image K at position (i, j) respectively.

[0121] (2)

[0122] Among them, μ I is the mean value of I, μ K is the mean value of K, is the variance of I, is the variance of K, σ IK is the covariance of I and K, c 1 =(k 1 L) 2 , c 2 =(k 2 L) 2 are two constants to avoid division by zero, L is the range of pixel values, i.e. 256, and the default value is k 1 =0.01, k 2 =0.03.

[0123] The experimental results are as follows Figure 6 As shown in (b), the transmission image restored by the A-CycleGAN method is natural and has clear texture details. The experimental results of the embodiment of the present invention and the traditional histogram threshold method on the artificial synthetic transmission image dataset are shown in Figure 2. Figure 7 As shown, the traditional histogram threshold method restores the result as follows Figure 7 As shown in (c), the effect after restoration is not ideal. The foreground document content is not fully restored and the content details are lost. The restoration results of the A-CycleGAN method are shown in Figure 7As shown in (b), the A-CycleGAN method can generate clearer details in both heavy and weak transmission areas without the need for paired data sets, and better restore the transmission document image. The experimental results of the embodiment of the present invention and the traditional histogram threshold method on the scanned transmission image obtained by the FS-1020MFP scanner are shown in Figure 2. Figure 8 As shown, the traditional histogram threshold method restores the result as follows Figure 8 As shown in (c), the restored result is blurry and the texture details are unclear. The restored result of A-CycleGAN method is as follows Figure 8 As shown in (b), the A-CycleGAN method can not only better remove background transmission content and noise, but also clearly preserve the text content.

[0124] Table 1 shows the comparison of objective evaluation standard parameters of the restoration results of the method of the present invention and the traditional histogram threshold method. Since there will be a certain degree of mixing inconsistency for scanning transmission images with different text contents when the front image and the back image are mixed, the scanning transmission image separation model has high robustness, which not only makes the performance of visual separation of a single image significant, but also maintains high restoration for large-scale image sets. By using the channel attention mechanism module and the regional attention mechanism module in the generator, the model can effectively process different features and regions, improve the responsiveness of the features, make the scanning transmission image separation effect more natural, the texture details clearer, and the color retention is better.

[0125] Table 1 Quantitative comparison of the results of different image restoration methods on artificial synthetic scanned transmission images

[0126]

[0127] Example 4

[0128] A scanning transmission image restoration device, see Figure 8 The device includes: a processor 1 and a memory 2, wherein the memory 2 stores program instructions, and the processor 1 calls the program instructions stored in the memory 2 to enable the device to execute the following method steps in Example 1:

[0129] The A-CycleGAN network model consisting of the first and second generators and the first and second discriminators is trained using a synthetic transmission image training set;

[0130] A first non-transmission image is generated and outputted by the first generator. If the first discriminator cannot recognize all the first non-transmission images, a first error is calculated by adversarial loss, and the parameters of the first discriminator are updated by back propagation to seek a first minimized classification error.

[0131] The first discriminator is fixed, and the first transmission image is generated and outputted through the second generator. The difference between the input synthetic transmission image and the first transmission image is calculated through the cycle consistency loss and the perceptual loss, and the first loss value is obtained. The network parameters of the first generator are updated through back propagation to seek the first global optimal value.

[0132] The second generator generates and outputs a second transmission image. If the second discriminator cannot recognize all the second transmission images, a second error is calculated by adversarial loss, and the parameters of the second discriminator are updated by back propagation to seek a second minimized classification error.

[0133] The second discriminator is fixed, and a second non-transmission image is generated and outputted through the first generator. The difference between the input non-transmission image and the second non-transmission image is calculated through the cycle consistency loss and the perceptual loss, and the second loss value is obtained. The network parameters of the second generator are updated through back propagation to seek the second global optimal value.

[0134] When all training is completed, the generator network outputs the final non-transmission image.

[0135] Among them, the A-CycleGAN network model includes:

[0136] Transmission-transmission branch: Use the input synthetic transmission image h to generate the first non-transmission image d through the first generator h , reconstructed into the first transmission image h′ through the second generator;

[0137] No-transmission-no-transmission branch: Use the input no-transmission image d to generate the second transmission image h through the second generator d , reconstructed into a second non-transmission image d′ through the first generator;

[0138] The two branches share the first and second generators, the transmission-transmission branch uses the first discriminator, and the no-transmission-no-transmission branch uses the second discriminator.

[0139] Furthermore, the first generator includes: a channel attention mechanism block and a regional attention mechanism block, and a skip connection structure is used to enable the high layer to obtain the image feature information of the low layer.

[0140] Among them, the channel attention mechanism block converts the feature map size from C×H×W to C×1×1, where C represents the number of channels and H×W represents the size of each channel feature map;

[0141] The processing process of the channel attention mechanism block is:

[0142]

[0143] Where pool(·) represents the pooling layer, Conv represents the convolutional layer, σ is the Sigmoid function, δ is the ReLU function, and F is the input of the channel attention mechanism block. is element-wise multiplication, F c Represents the output of the channel attention mechanism;

[0144] The regional attention mechanism block converts an image of shape C×H×W to 1×H×W. The processing process is:

[0145]

[0146] Among them, F result Represents the output of the region attention mechanism block.

[0147] Among them, the first generator is built using convolution, dense blocks and attention mechanism structures.

[0148] Furthermore, the dense block is used to extract all features from shallow to deep layers. The dense block consists of three groups of structural units, each of which includes: a structural block and an attention mechanism network.

[0149] Furthermore, the device uses unpaired images to perform scanning transmission image restoration tasks during network training.

[0150] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0151] The execution subjects of the above-mentioned processor 1 and memory 2 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0152] The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.

[0153] Example 5

[0154] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0155] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0156] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0157] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0158] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0159] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0160] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A scanning transmission image restoration method, characterized in that: The method comprises: The A-CycleGAN network model consisting of the first and second generators and the first and second discriminators is trained using a synthetic transmission image training set; A first non-transmission image is generated and outputted by the first generator. If the first discriminator cannot recognize all the first non-transmission images, a first error is calculated by adversarial loss, and the parameters of the first discriminator are updated by back propagation to seek a first minimized classification error. The first discriminator is fixed, and the first transmission image is generated and outputted through the second generator. The difference between the input synthetic transmission image and the first transmission image is calculated through the cycle consistency loss and the perceptual loss, and the first loss value is obtained. The network parameters of the first generator are updated through back propagation to seek the first global optimal value. The second generator generates and outputs a second transmission image. If the second discriminator cannot recognize all the second transmission images, a second error is calculated by adversarial loss, and the parameters of the second discriminator are updated by back propagation to seek a second minimized classification error. The second discriminator is fixed, and a second non-transmission image is generated and outputted through the first generator. The difference between the input non-transmission image and the second non-transmission image is calculated through the cycle consistency loss and the perceptual loss, and the second loss value is obtained. The network parameters of the second generator are updated through back propagation to seek the second global optimal value. When all training is completed, the generator network outputs the final non-transmission image.

2. A scanning transmission image restoration method according to claim 1, characterized in that: The A-CycleGAN network model includes: Transmission-transmission branch: Use the input synthetic transmission image h to generate the first non-transmission image d through the first generator h , reconstructed into the first transmission image h′ through the second generator; No-transmission-no-transmission branch: Use the input no-transmission image d to generate the second transmission image h through the second generator d , reconstructed into a second non-transmission image d′ through the first generator; The two branches share the first and second generators, the transmission-transmission branch uses the first discriminator, and the no-transmission-no-transmission branch uses the second discriminator.

3. A scanning transmission image restoration method according to claim 1, characterized in that: The first generator includes: a channel attention mechanism block and a regional attention mechanism block, and uses a jump connection structure to enable the high layer to obtain the image feature information of the low layer.

4. A scanning transmission image restoration method according to claim 3, characterized in that: The channel attention mechanism block converts the feature map size from C×H×W to C×1×1, where C represents the number of channels and H×W represents the size of each channel feature map; The processing process of the channel attention mechanism block is: Where pool(·) represents the pooling layer, Conv represents the convolutional layer, σ is the Sigmoid function, δ is the ReLU function, and F is the input of the channel attention mechanism block. is element-wise multiplication, F c Represents the output of the channel attention mechanism; The regional attention mechanism block converts an image of shape C×H×W to 1×H×W by: Among them, F result Represents the output of the region attention mechanism block.

5. A scanning transmission image restoration method according to claim 3, characterized in that: The first generator is built using convolution, dense blocks and attention mechanism structures. The dense block is used to extract all features from shallow to deep layers. The dense block is composed of three groups of structural units, and each group of structural units includes: a structural block and an attention mechanism network.

6. A scanning transmission image restoration method according to any one of claims 1 to 5, characterized in that: The method uses unpaired images to perform scanned transmission image restoration tasks during the network training process.

7. A scanning transmission image restoration device, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method steps described in claims 1-5.

Images