Fuzzy invoice image restoration method and device based on generative adversarial network

Abstract

The invention discloses a fuzzy invoice image restoration method and device based on a generative adversarial network, and the method comprises the steps: adding an attention gate module before the jump connection of an encoder and a decoder, and enabling the attention gate module to adjust the output features of the encoder again through adjusting the weights of features at different spatial positions, thereby achieving the restoration of a fuzzy invoice image. According to the method, the model learns which features are more concerned during feature fusion, so that important information is better reserved and unimportant information is better inhibited, the performance and generalization ability of the model are improved, the model generates a finer and more vivid invoice image, and compared with an original invoice image input to the model, the method has the advantages that the efficiency is improved, and the cost is reduced. And the definition of the invoice image generated by the model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for repairing blurred invoice images based on a generative adversarial network. Background Art

[0002] Invoices are evidence of a real transaction. Thanks to the unified portal provided by the state to verify the authenticity of invoices, invoices have become a strong support for the authenticity of trade backgrounds in bank supply chain financing credit products, greatly reducing financing risks. However, the authenticity and validity of invoice photocopies still need to be further confirmed. When banks implement supply chain financing solutions, the general practice is to first receive structured invoice information fields to quickly advance the financing process and realize loan release, and then supplement the invoice image and associate it with the previous invoice information structured fields, so as to facilitate the bank to carry out post-loan review. Therefore, how to accurately and efficiently associate the post-loan invoice image with the invoice structured fields during the loan becomes particularly important. In the early days, this operation was done manually. Bank staff checked the invoice number, invoice code and other information in the image, and the account manager reviewed and compared them, and then associated the invoice photocopies with the invoice structured information. With the development of science and technology, the banking industry has begun to use artificial intelligence technology to implement digital applications to liberate human labor. Taking the binding of invoice attachments as an example, a customer uploads an attachment package containing hundreds of photocopies of invoices, and uses OCR technology to recognize invoices, identifying and binding individual invoices one by one to the corresponding structured invoice information. Although this technical solution can reduce the manpower burden of banks to a certain extent, due to the inevitable problems of blurred images of invoices, insufficient light, and prominent noise, not every attachment in the invoice attachment package can be clearly identified, and the invoice structured information cannot be successfully bound to the invoice attachments.

[0003] Therefore, there is an urgent need to provide a technical solution to improve the clarity of invoice images so as to improve the accuracy and efficiency of subsequent invoice recognition. Summary of the invention

[0004] The present application provides a method and device for repairing blurred invoice images based on a generative adversarial network, which can improve the clarity of generated invoice images. The technical solution is as follows.

[0005] In a first aspect, a method for repairing blurred invoice images based on a generative adversarial network is provided, the method comprising:

[0006] Inputting the blurred image to be repaired into a generator in a generative adversarial network, the generator includes an attention gate module, an encoder and a decoder, the encoder includes a first LeWin transformer block, the decoder includes a second LeWin transformer block and a third LeWin transformer block, there is a jump connection between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, the attention gate module is arranged before the jump connection between the encoder and the decoder, and the blurred image is an invoice image;

[0007] Encoding the features of the blurred image by using the first LeWin transformer block in the encoder to obtain a first feature map of the blurred image;

[0008] Decoding the first feature map of the blurred image by the second LeWin transformer block in the decoder to obtain a second feature map of the blurred image;

[0009] Performing feature fusion on the first feature map and the second feature map through the attention gate module to obtain a third feature map, generating an attention map based on the third feature map, and weighting the third feature map based on the attention map to obtain a fourth feature map, wherein the attention map is used to characterize the importance of each position in the third feature map;

[0010] The third LeWin transformer block in the decoder performs decoding based on the second feature map and the fourth feature map to obtain a restored image.

[0011] In some implementations, the attention gate module includes a convolution layer, an upsampling layer, and an activation function. Before the first feature map and the second feature map are feature-fused by the attention gate module, the method further includes:

[0012] Convolving the first feature map and the second feature map respectively through the convolution layer in the attention gate module, so that the number of channels of the first feature map is adjusted to be consistent with the number of channels of the second feature map;

[0013] The performing feature fusion on the first feature map and the second feature map by the attention gate module includes:

[0014] Upsampling the first feature map after convolution and the second feature map after convolution respectively;

[0015] The upsampled first feature map and the upsampled second feature map are added, and the added image is subjected to nonlinear transformation through an activation function to obtain a third feature map;

[0016] The generating an attention map based on the third feature map comprises:

[0017] The attention score of each position in the third feature map is determined by a convolution layer and an activation function to obtain an attention map, where the attention score is used to characterize the importance of the corresponding position.

[0018] In some embodiments, the generative adversarial network further includes a discriminator, and before inputting the blurred image to be repaired into the generator in the generative adversarial network, the method further includes:

[0019] Performing fuzzy processing on the sample real image to obtain a sample fuzzy image, wherein the sample real image is an invoice image;

[0020] Inputting the sample blurred image into the generator to obtain a sample repaired image;

[0021] The sample repaired image and the sample real image are input into the discriminator, and the sample repaired image is cropped by the discriminator to obtain a plurality of sample repaired image blocks, and the sample real image is cropped to obtain a plurality of sample real image blocks; feature extraction is performed on each of the sample real image blocks to obtain the features of each of the sample real image blocks, and feature extraction is performed on each of the sample repaired image blocks to obtain the features of each of the sample repaired image blocks; based on the features of each of the sample real image blocks and the features of each of the sample repaired image blocks, the distance between each of the sample repaired image blocks and the corresponding sample real image blocks is determined; the distance between each of the sample repaired image blocks and the corresponding sample real image blocks is averaged to obtain the adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image and the sample real image discriminated by the discriminator;

[0022] Determining an overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image;

[0023] The parameters of the generator and the parameters of the discriminator are adjusted based on the overall loss value of the generative adversarial network.

[0024] In some implementations, determining the overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image includes:

[0025] Performing feature extraction on the sample repaired image and the sample real image respectively to obtain a feature map of the sample repaired image and a feature map of the sample real image;

[0026] Determine a content loss value of the sample repaired image based on the feature map of the sample repaired image and the feature map of the sample real image, wherein the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image;

[0027] Based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image, an overall loss value of the generative adversarial network is determined.

[0028] In some implementations, the extracting features from the sample restored image and the sample real image respectively includes:

[0029] Using multiple layers of a pre-trained VGG network to perform feature extraction on the sample repaired image and the sample real image respectively, so as to obtain multiple feature maps of the sample repaired image and multiple feature maps of the sample real image output by the multiple layers;

[0030] The determining, based on the feature map of the sample inpainted image and the feature map of the sample real image, a content loss value of the sample inpainted image comprises:

[0031] For a feature map of the sample inpainted image output by any layer of the multiple layers of the VGG network, determine a mean square error loss between the feature map of the sample inpainted image and a feature map of the sample real image output by the same layer in the VGG network;

[0032] The sum of the mean square error losses corresponding to each feature map in the plurality of feature maps of the sample repaired image is determined to obtain a content loss value of the sample repaired image.

[0033] In some implementations, determining the overall loss value of the generative adversarial network based on the adversarial loss value of the sample inpainted image and the content loss value of the sample inpainted image includes:

[0034] A weighted sum is performed on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image to obtain an overall loss value of the generative adversarial network.

[0035] In some embodiments, cropping the sample repaired image includes: cropping a random number of image blocks of random sizes from random positions in the sample repaired image;

[0036] The cropping of the sample real image comprises: cropping a random number of image blocks of random sizes from random positions in the sample real image;

[0037] The determining of the distance between each of the sample restored image blocks and the corresponding sample real image blocks comprises:

[0038] For one of the sample restored image blocks, determining a target sample restored image block having the same position and the same size as the sample restored image block from a plurality of sample real image blocks cropped from the sample real image;

[0039] A Wasserstein distance between the sample inpainted image block and a target sample inpainted image block is determined.

[0040] In a second aspect, a device for repairing blurred invoice images based on a generative adversarial network is provided, the device comprising:

[0041] An input unit, used for inputting a blurred image to be repaired into a generator in a generative adversarial network, wherein the generator includes an attention gate module, an encoder and a decoder, wherein the encoder includes a first LeWin transformer block, and the decoder includes a second LeWin transformer block and a third LeWin transformer block, wherein a jump connection exists between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, and the attention gate module is arranged before the jump connection between the encoder and the decoder, and the blurred image is an invoice image;

[0042] an encoder, configured to encode the features of the blurred image by using the first LeWin transformer block to obtain a first feature map of the blurred image;

[0043] a decoder, configured to decode the first feature map of the blurred image through the second LeWin transformer block to obtain a second feature map of the blurred image;

[0044] an attention gate module, configured to perform feature fusion on the first feature map and the second feature map to obtain a third feature map, generate an attention map based on the third feature map, and weight the third feature map based on the attention map to obtain a fourth feature map, wherein the attention map is used to characterize the importance of each position in the third feature map;

[0045] The decoder is used to decode based on the second feature map and the fourth feature map through the third LeWin transformer block to obtain a restored image.

[0046] In some embodiments, the attention gate module includes a convolution layer, an upsampling layer, and an activation function. Before the attention gate module performs feature fusion on the first feature map and the second feature map, the attention gate module is also used to convolve the first feature map and the second feature map respectively through the convolution layer to adjust the number of channels of the first feature map to be consistent with the number of channels of the second feature map; upsample the convolved first feature map and the convolved second feature map respectively; add the upsampled first feature map and the upsampled second feature map, and perform a nonlinear transformation on the added image through the activation function to obtain a third feature map; determine the attention score of each position in the third feature map through the convolution layer and the activation function to obtain an attention map, and the attention score is used to characterize the importance of the corresponding position.

[0047] In some embodiments, the generative adversarial network further includes a discriminator, and the device further includes: a preprocessing unit, configured to perform blurring processing on the sample real image to obtain a sample blurred image, wherein the sample real image is an invoice image;

[0048] The input unit is further used to input the sample blurred image into the generator to obtain a sample repaired image; and input the sample repaired image and the sample real image into the discriminator;

[0049] The discriminator is used to crop the sample repaired image to obtain a plurality of sample repaired image blocks, and to crop the sample real image to obtain a plurality of sample real image blocks; perform feature extraction on each of the sample real image blocks to obtain the features of each of the sample real image blocks, and perform feature extraction on each of the sample repaired image blocks to obtain the features of each of the sample repaired image blocks; determine the distance between each of the sample repaired image blocks and the corresponding sample real image blocks based on the features of each of the sample real image blocks and the features of each of the sample repaired image blocks; perform an average operation on the distance between each of the sample repaired image blocks and the corresponding sample real image blocks to obtain an adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image and the sample real image discriminated by the discriminator;

[0050] The apparatus further comprises: a loss determination unit, configured to determine an overall loss value of the generative adversarial network based on the adversarial loss value of the sample inpainted image;

[0051] A parameter adjustment unit is used to adjust the parameters of the generator and the parameters of the discriminator based on the overall loss value of the generative adversarial network.

[0052] In some embodiments, the loss determination unit is used to perform feature extraction on the sample repaired image and the sample real image, respectively, to obtain a feature map of the sample repaired image and a feature map of the sample real image; determine a content loss value of the sample repaired image based on the feature map of the sample repaired image and the feature map of the sample real image, wherein the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image; determine an overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image.

[0053] In some embodiments, the loss determination unit is used to use multiple layers of a pre-trained VGG network to perform feature extraction on the sample repaired image and the sample real image respectively, so as to obtain multiple feature maps of the sample repaired image output by the multiple layers and multiple feature maps of the sample real image; for the feature map of the sample repaired image output by any layer of the multiple layers of the VGG network, determine the mean square error loss between the feature map of the sample repaired image and the feature map of the sample real image output by the same layer in the VGG network; determine the sum of the mean square error losses corresponding to each feature map in the multiple feature maps of the sample repaired image, so as to obtain the content loss value of the sample repaired image.

[0054] In some embodiments, the loss determination unit is used to perform a weighted sum of the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image to obtain an overall loss value of the generative adversarial network.

[0055] In some embodiments, the discriminator is used to crop a random number of image blocks of random sizes from random positions in the sample repaired image; crop a random number of image blocks of random sizes from random positions in the sample real image; for one of the sample repaired image blocks, determine a target sample repaired image block with the same position and size as the sample repaired image block from multiple sample real image blocks cropped from the sample real image; and determine the Wasserstein distance between the sample repaired image block and the target sample repaired image block.

[0056] In a third aspect, a computer device is provided, the computer device comprising a processor, the processor being coupled to a memory, the memory storing at least one computer program instruction, the at least one computer program instruction being loaded and executed by the processor, so that the computer device implements the method provided in the first aspect or any optional manner of the first aspect. The specific details of the computer device provided in the third aspect can be found in the first aspect or any optional manner of the first aspect, and will not be repeated here.

[0057] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium. When the instruction is executed on a computer, the computer executes the method provided by the first aspect or any optional manner of the first aspect.

[0058] In a fifth aspect, a computer program product is provided, which includes one or more computer program instructions. When the computer program instructions are loaded and executed by a computer, the computer executes the method provided by the first aspect or any optional manner of the first aspect.

[0059] In the embodiment of the present application, an attention gate module is added before the jump connection between the encoder and the decoder. The attention gate module readjusts the output features of the encoder by adjusting the weights of features at different spatial positions, so that the model learns which features to pay more attention to during feature fusion, thereby better retaining important information and suppressing unimportant information. This helps to improve the performance and generalization ability of the model, allowing the model to generate more delicate and realistic invoice images. Compared with the original invoice images input to the model, the clarity of the invoice images generated by the model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is an architecture diagram of a generative adversarial network provided in an embodiment of the present application;

[0061] Figure 2 It is a schematic diagram of the architecture of a generator network model provided in an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of the architecture of an attention gating signal module provided in an embodiment of the present application;

[0063] Figure 4 It is a schematic diagram of the architecture of a discriminator network model provided in an embodiment of the present application;

[0064] Figure 5 It is a flowchart diagram of a method for repairing blurred invoice images based on a generative adversarial network provided in an embodiment of the present application;

[0065] Figure 6It is a structural schematic diagram of a blurred invoice image repair device based on a generative adversarial network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0067] The embodiment of the present application uses a discriminator to verify the authenticity of the image generated by the generator. In a series of confrontation processes, the ability of the generator is continuously optimized, thereby improving the fidelity of the generated image. The embodiment of the present application has made many improvements to the network structure. First, the embodiment of the present application uses the U-Net structure as the generator because the U-Net structure can achieve a better balance between image reconstruction performance, computational cost and time cost. Secondly, the embodiment of the present application uses PathGAN as the discriminator network structure, which is based on the Pix2pix framework. Unlike traditional discriminators, PathGAN divides the input image (such as a real image as a sample and a repaired image generated by the generator as a sample) into multiple NxN image blocks (such as by cropping), and determines the distance between each image block in the generated repair image and the corresponding image block in the real image, and then averages the distance of each image block to obtain a judgment on the entire repaired image. This method not only makes the judgment more authoritative, but also helps the generator to better learn local details, thereby generating more delicate and realistic invoice images and improving image clarity.

[0068] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0069] 1. Generative Adversarial Network (GAN): Generative Adversarial Network is a machine learning model that includes a generator and a discriminator. The generator tries to generate data that looks like real samples, while the discriminator tries to distinguish between real samples and samples generated by the generator. This competition-driven learning process can help the generator generate more realistic data.

[0070] 2. Generator: The part of the generative adversarial network that is responsible for generating samples that look like real data. The generator improves its generative ability by learning techniques for generating data from random noise.

[0071] 3. Discriminator: Another part of the generative adversarial network, responsible for distinguishing between the data generated by the generator and the real data. The discriminator improves its ability to distinguish between the two types of data by learning.

[0072] 4.UNet: UNet is a deep learning architecture for image segmentation. It contains symmetrical encoder and decoder parts, which can effectively process the input image and output the segmentation mask for each pixel.

[0073] 5. PathGAN: PathGAN is a variant of a generative adversarial network for generating medical images. It introduces path rules in the generator and discriminator to help generate more realistic and interpretable medical images.

[0074] 6. Loss function: The loss function measures the difference between the model's predicted output and the actual label. During training, the optimization algorithm tries to minimize the loss function so that the model's predictions are closer to the true values.

[0075] 7. Self-attention mechanism: The self-attention mechanism is an attention mechanism for processing sequence data, which allows the model to take into account the information of other positions in the sequence when calculating the representation of each position. This mechanism is often used in natural language processing and image processing tasks.

[0076] 8.AG module: The AG module is a module for image processing that combines the generative adversarial network and the self-attention mechanism. It can generate high-quality images and use the self-attention mechanism in the image generation process to improve the generation effect.

[0077] 9.GELU activation function: GELU is an activation function, the full name is Gaussian Error Linear Unit. It is used to introduce nonlinearity in deep learning and is usually used to improve model performance and training speed.

[0078] The following is an example of an application scenario of the embodiment of the present application.

[0079] Image restoration can be viewed as a problem of converting a blurry image to a clear image. Generative adversarial networks (GANs) have become a revolutionary technology in the field of image restoration. The mainstream GAN framework uses the U-Net structure as a generator, PathGAN as a discriminator, and a combined loss function to measure the difference between the output and the target.

[0080] GAN can not only preserve the texture details of the image, but also produce outputs that are highly similar to the original image. Therefore, it is more appropriate to apply this technology to the scenario of improving the clarity of invoices. However, there are still some problems with traditional GAN ​​technology that cannot be ignored. In the scenario set in the embodiment of the present application, the invoice images are uploaded by the customer themselves, and the server cannot predict the number of invoices and the size of each invoice, so there are high requirements for the processing speed and stability of the algorithm. However, as the network deepens, although traditional GAN ​​technology can obtain richer global information, it will inevitably increase the number of parameters, thereby increasing the computational burden and running time. Furthermore, the instability of the target size often makes small targets after deblurring become blurred or mistakenly removed, thereby losing the details and contextual information of the image.

[0081] The embodiment of the present application upgrades and optimizes the image restoration model of the entire generative adversarial network from three aspects: generator, discriminator and loss function.

[0082] First, the generator uses a relatively simple network structure unit, LeWin Transformer Block, and adds a module called "attention gate" before the jump connection between the encoder and decoder to ensure the performance and efficiency of the network while reducing the computational complexity of the generator. The LeWin Transformer Block helps to obtain richer global information without increasing the number of parameters, thereby avoiding the increase in computational burden and running time caused by the increase in the number of parameters.

[0083] Secondly, a dual-scale discriminator is used to capture the local texture and details of the image while taking into account the grasp of the global structure and content information.

[0084] Thirdly, a method combining content loss and adversarial loss is proposed to recalculate the loss function. By reasonably allocating the weights of these two losses, the color and texture distortions in the generated images can be better corrected.

[0085] The method proposed in the embodiment of this application effectively simplifies the network structure of the generator and strengthens the discriminator network, using a simpler network to produce higher quality invoice images, while reducing the computational burden of the model and improving the model accuracy. Through this series of improvements and optimizations, a more efficient and accurate invoice image restoration model INVOICE-NAFGAN is constructed, which can output higher quality invoice images while maintaining computational efficiency and reducing server burden.

[0086] In addition, the instability of the target size often makes the small targets after deblurring become blurred or mistakenly removed, thereby losing the details and context information of the image. In the embodiment of the present application, since the discriminator crops the image into multiple image blocks, the multiple image blocks are compared separately to determine the adversarial loss, and small targets are cut into the image blocks as much as possible, thereby increasing the probability of small targets participating in the loss calculation and reducing the impact of the removal of small targets after deblurring on the accuracy of the model.

[0087] Please refer to the attached Figure 1 , attached Figure 1It is an architecture diagram of a generative adversarial network provided in an embodiment of the present application. The generative adversarial network includes a generator and a discriminator. The generator is used to repair blurred images, such as converting an invoice image from a blurred image to a clear image. The discriminator is used to verify the authenticity of the image generated by the generator. In a series of adversarial processes, the ability of the generator is continuously optimized, thereby improving the fidelity of the generated image. The embodiment of the present application has made many improvements to the network structure. First, the embodiment of the present application adopts the U-Net structure as the generator because it can achieve a better balance between image reconstruction performance, computational cost and time cost. Secondly, the embodiment of the present application selects PathGAN as the discriminator network structure, which is based on the Pix2pix framework. Unlike traditional discriminators, PathGAN divides the input image (sample repaired image and sample real image) into multiple NxN image blocks, evaluates the distance between each image block in the sample repaired image and the corresponding image block in the sample real image, and then averages the distance of each image block to obtain a judgment on the entire sample repaired image. This method not only makes the judgment more authoritative, but also helps the generator to better learn local details, thereby generating more delicate and realistic invoice images and improving image clarity.

[0088] In order to train the network more effectively, the embodiment of the present application adopts a combined loss function that combines adversarial loss and content loss. This loss strategy aims to achieve high consistency at the pixel level while maintaining authenticity and coherence in higher-level feature spaces. By integrating the advantages of the U-shaped network structure of U-Net and PathGAN, the embodiment of the present application proposes an improved GAN framework that aims to effectively address the problem of invoice image restoration.

[0089] The embodiment of the present application draws on the LeWin Transformer Block as the core component of the generator network, and refers to the Uformer structure as the basic framework of the generator network. This is because the Uformer structure combines the characteristics of the U-Net structure that can fully integrate shallow features and deep features, as well as the self-attention mechanism of the Transformer, providing a powerful and efficient method for understanding and generating images. The embodiment of the present application optimizes the Uformer structure, reduces the number of parameters and introduces a more efficient attention mechanism, so that it is easier to integrate it into the overall architecture of the network model of the embodiment of the present application, reduces the model calculation overhead, and the most important goal is to improve the sensitivity and prediction accuracy of the model.

[0090] Please refer to the attached Figure 2 , attached Figure 2 It is a schematic diagram of the architecture of a generator network model provided in an embodiment of the present application.

[0091] The generator consists of multiple attention gate modules, encoders and decoders. The network structure of the generator is U-shaped. Figure 2 The left side is the encoder in the generator, and the Figure 2 The right side of the figure is the decoder in the generator, and there is a skip connection between the encoder and the decoder. Before skip-connecting the encoder's resolution features with the decoder's corresponding features, the present embodiment introduces an attention gate module to readjust the encoder's output features.

[0092] For example, Figure 2 As shown in the attached Figure 2 The encoder includes a LeWin transformer block 210, and the decoder includes a LeWin transformer block 220 and a LeWin transformer block 230. The LeWin transformer block 210 is jump-connected to the LeWin transformer block 230, and the attention gate module 300 is arranged before the jump connection between the LeWin transformer block 210 and the LeWin transformer block 230. The input end of the attention gate module 300 is connected to the LeWin transformer block 210, and in addition, the LeWin transformer block 220 is connected to the input end of the attention gate module 300 after passing through a 2-fold upsampling layer. The output end of the attention gate module 300 is connected to the LeWin transformer block 230.

[0093] The attention gate module generates a gating signal to control the weights of features at different spatial locations, such as Figure 2 As shown in the circle in . The jump connection between the encoder and decoder is implemented by direct splicing.

[0094] Specifically, the attention gate module takes the feature map (dimension is Hereinafter referred to as the first feature map) and the feature map from the output of the decoder (dimension is The first feature map (hereinafter referred to as the second feature map) is spliced ​​together in the channel dimension to obtain an output with a dimension of C*H*W (hereinafter referred to as the third feature map). In order to more accurately adjust the weights of features at different spatial positions, the embodiment of the present application introduces a module called "Attention Gate" (AG) before the jump connection between the encoder and the decoder. The AG module includes a convolution layer, an upsampling layer, and an activation function.

[0095] Taking the AG module 300 as an example, the working principle and structural features of the AG module are illustrated below. After the blurred image is encoded by the LeWin transformer block 202 in the encoder, it is input into the LeWin transformer block 210 in the encoder after convolution and downsampling. The LeWin transformer block 210 in the encoder encodes the features of the blurred image to obtain a feature map x (first feature map) of the blurred image, and the LeWin transformer block 220 in the decoder encodes the feature map x of the blurred image to obtain a feature map z of the blurred image. The feature map z of the blurred image is downsampled by 2 times to obtain a feature map g (second feature map) of the blurred image.

[0096] Step 1. Channel matching.

[0097] The feature map x (first feature map) and feature map g (second feature map) of the blurred image input to the attention gate module 300 first pass through a 1x1 convolution layer contained in the attention gate module 300. The purpose of the convolution is to adjust the number of channels of the feature map x and the number of channels of the feature map g to the same number while keeping the spatial dimensions of the feature map x and the feature map g unchanged.

[0098] Step 2. Feature fusion.

[0099] After the upsampling operation is completed (at this time, the feature map x and the feature map g have the same spatial dimension), the upsampled feature map x and the upsampled feature map g are added, and the added image is nonlinearly transformed through the ReLU activation function to obtain the feature map k (hereinafter referred to as the third feature map).

[0100] Step 3. Attention score calculation.

[0101] The attention score of each position in feature map k (the third feature map) is calculated through another 1x1 convolution layer and a sigmoid activation function to obtain an attention map. The value of each position in the attention map is an attention score. The attention score is an importance score between 0 and 1. The attention score of a position in the attention map is used to indicate the importance of the position in the feature map.

[0102] Step 4. Attention weighting.

[0103] The feature map k (third feature map) fused in step 2 is weighted using the calculated attention map, for example, by multiplying the attention score of each position in the attention map with the pixel value of the corresponding position in the third feature map to obtain a fourth feature map, which is the final output of the attention gate module 300.

[0104] By introducing the AG module, the generator can learn which features to pay more attention to when merging features, thereby better retaining important information and suppressing unimportant information. This helps improve the performance and generalization ability of the model. The structure of the attention gating signal module is as follows Figure 3 shown.

[0105] Discriminator network

[0106] In a generative adversarial network (GAN), the main task of the discriminator is to evaluate the data distribution difference between the real clear image and the generator output image, with the goal of narrowing the distance between the two to drive the generator to generate more realistic clear images. To achieve this goal, the discriminator needs to accurately evaluate the differences in the overall structure, content, and detailed texture information of the image. Based on these considerations, the present application embodiment selects PatchGAN as the network structure of the discriminator.

[0107] Its advantages include:

[0108] 1. Highly flexible discrimination target: PatchGAN can process input images of different sizes without restrictions, greatly improving the generalization ability of the network.

[0109] 2. Improve discrimination quality and training efficiency: Through block discrimination and average confidence calculation methods, PatchGAN can more accurately identify true and false images, thereby improving the quality of the discriminator and speeding up the network training.

[0110] 3. Deep image texture analysis: Since PatchGAN makes a distinction on each small patch, it can analyze the local texture information of the image more deeply. This not only conforms to the image reconstruction theory of the embodiment of this application, emphasizing the importance of overall content and structure, but also allows for a deeper level of image detail texture analysis. This means that the generator can more effectively simulate the data distribution of the real image domain, thereby producing higher quality images.

[0111] By introducing Wasserstein GAN (WGAN), the present application embodiment hopes to build a more stable and reliable generative adversarial network model. WGAN can improve the training stability and output diversity of GAN. The UformerGAN model discriminator network model proposed in the present application embodiment is as follows Figure 4 As shown in Figure 2. The network is designed to effectively discriminate between real and generated images, thereby guiding the generator network to produce higher quality outputs.

[0112] Exemplarily, for each sample real image in the sample image set, the sample real image is blurred to obtain a sample blurred image, where the sample real image is an invoice image; the sample blurred image is input into the generator to obtain a sample repaired image. The sample repaired image generated by the generator and the sample real image are input into the discriminator, and the method flow executed by the discriminator is as follows.

[0113] Step 1. Input and preprocessing: The discriminator crops the sample repaired image to obtain multiple sample repaired image blocks, and crops the sample real image to obtain multiple sample real image blocks.

[0114] In some embodiments, the discriminator randomly crops the input image to increase the robustness and diversity of the network. For example, the discriminator crops a random number of image patches of random size from random positions in the sample repaired image; the discriminator crops a random number of image patches of random size from random positions in the sample real image.

[0115] Step 2. Perform feature extraction on multiple sample repaired image blocks and multiple sample real image blocks respectively through the first convolutional layer.

[0116] Feature extraction includes convolution and activation.

[0117] For the convolution process, the image blocks cropped from the sample repaired image and the image blocks cropped from the sample real image are respectively input into a convolution layer with a convolution kernel size of 4x4. Through the convolution layer, the multiple image blocks cropped from the sample repaired image and the multiple image blocks cropped from the sample real image are respectively convolved to obtain the features of each sample repaired image block and the features of each real repaired image block.

[0118] For the activation process, the GELU activation function is used to perform nonlinear calculations on the features obtained by block convolution of each sample repaired image and the features obtained by block convolution of each real repaired image.

[0119] Step 3. Further feature extraction is performed on each sample repaired image block and each real repaired image block through the intermediate convolution block. The convolution block includes a convolution layer, a normalization layer, and an activation layer.

[0120] Convolutional layer: After the GELU activation layer, three sets of identical convolutional blocks are set, each of which starts with a convolutional layer with a 4x4 convolution kernel size for further feature extraction.

[0121] Normalization layer: The convolution layer is followed by a normalization layer, which helps maintain the stability and convergence speed of network training.

[0122] Activation layer: Each convolution block ends with a GELU activation layer to introduce nonlinearity and further extract important features.

[0123] Step 4. Final convolutional layer: After three sets of convolutional blocks, the features of each sample repaired image block and each true repaired image block will enter the final 4x4 convolutional layer for deeper feature extraction.

[0124] Step 5. Based on the characteristics of each sample real image block and the characteristics of each sample repaired image block, determine the distance between each sample repaired image block and the corresponding sample real image block; average the distance between each sample repaired image block and the corresponding sample real image block to obtain the adversarial loss value of the sample repaired image, which is used to characterize the difference between the sample repaired image discriminated by the discriminator and the sample real image.

[0125] In a possible implementation, the embodiment of the present application uses Wasserstein distance to measure the restoration effect of the image, which provides a more stable and effective optimization target.

[0126] For a sample repaired image block, determine a target sample repaired image block with the same position and size as the sample repaired image block from multiple sample real image blocks cropped from the sample real image; determine the Wasserstein distance between the sample repaired image block and the target sample repaired image block. Average the Wasserstein distance of each sample real image block in the multiple sample real image blocks to obtain the adversarial loss value of the sample repaired image.

[0127] In this way, the PatchGAN discriminator network adopted in the embodiments of the present application can effectively distinguish between real and generated images, thereby helping to guide the generator network to produce higher quality outputs.

[0128] Model loss function

[0129] In an embodiment of the present application, the embodiment of the present application constructs a loss function under the framework of a generative adversarial network, aiming to ensure that the generated repaired image is consistent with the original image in content and can be properly adversarially trained with the discriminator. To achieve this goal, the embodiment of the present application divides the loss function into two key parts. The first part is the content loss, which aims to ensure that the generated repaired image is consistent with the original input blurred image in content; the second part is the adversarial loss to meet the network's requirements for the discriminator. Through the organic combination of these two parts, this study is committed to promoting the generator to generate better quality repaired images.

[0130] Fighting Losses

[0131] The design purpose of the adversarial loss is to ensure that the restored image output by the generator is consistent with the original blurred image in content, while better approaching the data distribution of the real image. The embodiment of the present application adopts the WGAN-GP loss function to ensure the stability of the training process and effectively avoid the problem of gradient disappearance or explosion, so that the generator can learn the data distribution more efficiently. This enables the embodiment of the present application to achieve more stable and continuous gradient updates during the training process, thereby generating higher quality restored images.

[0132] For example, the specific adversarial loss function is shown in Formula 1.

[0133]

[0134] G: Generator, the task of the generator is to generate a d Generate the repaired image.

[0135] I d : The blurred image input to the generator G.

[0136] G(I d ): The generator G receives the blurred image I d As input, a repaired image is generated and output.

[0137] D: Discriminator, the task of the discriminator is to distinguish whether an image is generated by the generator or comes from a real clear image dataset.

[0138] D(G(I d )): The discriminator D receives the repaired image produced by the generator G as input and outputs a loss value (also called a score) to indicate whether the image is real or generated by the generator.

[0139] L adv : Adversarial loss function, which measures the difference between the inpainted images generated by the generator and the real images, is calculated by summing the outputs of the discriminator.

[0140] N: number of images in a batch. A batch is, for example, a set of sample images. The generator generates a sample repaired image for each sample blurred image in the sample image set. The embodiment of the present application calculates the loss for each sample repaired image in the batch separately, and adds the losses of all repaired images in a batch as the total loss.

[0141] Content loss

[0142] In deep learning, content loss plays a vital role. Content loss is responsible for ensuring that the content of the repaired image generated by the generator is consistent with the content of the original blurred image. In some embodiments, feature extraction is performed on the sample repaired image and the sample real image respectively to obtain a feature map of the sample repaired image and a feature map of the sample real image; based on the feature map of the sample repaired image and the feature map of the sample real image, a content loss value of the sample repaired image is determined, and the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image.

[0143] The embodiment of the present application adopts perceptual loss as content loss. The determination method of perceptual loss has achieved remarkable results in image generation tasks such as style transfer, super-resolution and image restoration. The core concept of content loss (perceptual loss) is to use the pre-trained VGG network to extract the feature map of the image and measure the difference between the two images by calculating the mean square error loss.

[0144] Exemplarily, multiple layers of a pre-trained VGG network are used to perform feature extraction on the sample repair image and the sample real image respectively, so as to obtain multiple feature maps of the sample repair image output by multiple layers and multiple feature maps of the sample real image; for the feature map of the sample repair image output by any layer of the multiple layers of the VGG network, the mean square error loss between the feature map of the sample repair image and the feature map of the sample real image output by the same layer in the VGG network is determined; the sum of the mean square error losses corresponding to each feature map in the multiple feature maps of the sample repair image is determined to obtain the content loss value of the sample repair image. For example, the specific form of the loss function is shown in Formula 2:

[0145]

[0146] In this formula:

[0147] y1 is the image generated by the generator (sample inpainted image).

[0148] 2 is the real image (sample real image).

[0149] Φ i Represents the feature map extraction function of the i-th layer of the VGG19 network.

[0150] Wi , H i , C i are the width, height and number of channels of the feature map of the i-th layer respectively.

[0151] The mean square error of the layer is calculated by summing all widths w, heights h, and channels c.

[0152] This perceptual loss function guides the generator to generate images that are more similar to the real images in higher-level feature representations, rather than just at the pixel level, thereby producing better results.

[0153] The overall loss function of the generative adversarial network

[0154] In some embodiments, the overall loss of the generative adversarial network is obtained by weighted summing the adversarial loss and the content loss. For example, the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image are weighted summed to obtain the overall loss value of the generative adversarial network.

[0155] Exemplarily, the overall loss function of the generative adversarial network is shown in Formula 3.

[0156] L total =λ·L adv +L content (3)

[0157] λ is a weight parameter. Given that the goal of the embodiment of the present application is to repair blurred images and reconstruct higher quality clear images, the model will focus more on minimizing content loss rather than adversarial loss. This helps ensure that the generated image is closer to the real image in terms of content.

[0158] Figure 5 This is a flowchart of a method for repairing blurred invoice images based on a generative adversarial network provided in an embodiment of the present application, the method comprising:

[0159] S510, inputting the blurred image to be repaired into a generator in a generative adversarial network, the generator including an attention gate module, an encoder and a decoder, the encoder including a first LeWin transformer block, the decoder including a second LeWin transformer block and a third LeWin transformer block, there is a jump connection between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, the attention gate module is arranged before the jump connection between the encoder and the decoder, and the blurred image is an invoice image;

[0160] S520, encoding the features of the blurred image through a first LeWin transformer block in the encoder to obtain a first feature map of the blurred image;

[0161] S530, decoding the first feature map of the blurred image through a second LeWin transformer block in the decoder to obtain a second feature map of the blurred image;

[0162] S540, performing feature fusion on the first feature map and the second feature map through an attention gate module to obtain a third feature map, generating an attention map based on the third feature map, and weighting the third feature map based on the attention map to obtain a fourth feature map, where the attention map is used to characterize the importance of each position in the third feature map;

[0163] S550, performing decoding based on the second feature map and the fourth feature map through a third LeWin transformer block in the decoder to obtain a restored image.

[0164] In some embodiments, the attention gate module includes a convolution layer, an upsampling layer, and an activation function. Before the first feature map and the second feature map are feature-fused by the attention gate module, the method further includes:

[0165] Convolving the first feature map and the second feature map respectively through the convolution layer in the attention gate module, so that the number of channels of the first feature map is adjusted to be consistent with the number of channels of the second feature map;

[0166] The first feature map and the second feature map are subjected to feature fusion through the attention gate module, including:

[0167] Upsampling the first feature map after convolution and the second feature map after convolution respectively;

[0168] The upsampled first feature map and the upsampled second feature map are added, and the added image is subjected to nonlinear transformation through an activation function to obtain a third feature map;

[0169] Generate an attention map based on the third feature map, including:

[0170] The attention score of each position in the third feature map is determined by the convolution layer and the activation function to obtain the attention map. The attention score is used to characterize the importance of the corresponding position.

[0171] In some embodiments, the generative adversarial network further includes a discriminator, and before the blurred image to be repaired is input into the generator in the generative adversarial network, the method further includes:

[0172] Performing fuzzy processing on the sample real image to obtain a sample fuzzy image, where the sample real image is an invoice image;

[0173] Input the sample blurred image into the generator to obtain a sample repaired image;

[0174] The sample repaired image and the sample real image are input into the discriminator, and the sample repaired image is cropped by the discriminator to obtain a plurality of sample repaired image blocks, and the sample real image is cropped to obtain a plurality of sample real image blocks; feature extraction is performed on each sample real image block to obtain the feature of each sample real image block, and feature extraction is performed on each sample repaired image block to obtain the feature of each sample repaired image block; based on the feature of each sample real image block and the feature of each sample repaired image block, the distance between each sample repaired image block and the corresponding sample real image block is determined; the distance between each sample repaired image block and the corresponding sample real image block is averaged to obtain the adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image discriminated by the discriminator and the sample real image;

[0175] Determine the overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image;

[0176] The parameters of the generator and the discriminator are adjusted based on the overall loss value of the generative adversarial network.

[0177] In some implementations, determining an overall loss value of a generative adversarial network based on an adversarial loss value of a sample repaired image includes:

[0178] Performing feature extraction on the sample repaired image and the sample real image respectively to obtain a feature map of the sample repaired image and a feature map of the sample real image;

[0179] Based on the feature map of the sample repaired image and the feature map of the sample real image, determine the content loss value of the sample repaired image, where the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image;

[0180] Based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image, the overall loss value of the generative adversarial network is determined.

[0181] In some implementations, feature extraction is performed on the sample repaired image and the sample real image respectively, including:

[0182] Using multiple layers of a pre-trained VGG network to perform feature extraction on the sample repaired image and the sample real image respectively, so as to obtain multiple feature maps of the sample repaired image and multiple feature maps of the sample real image output by the multiple layers;

[0183] Based on the feature map of the sample repaired image and the feature map of the sample real image, the content loss value of the sample repaired image is determined, including:

[0184] For a feature map of a sample inpainted image output by any of the multiple layers of the VGG network, determine the mean square error loss between the feature map of the sample inpainted image and the feature map of the sample real image output by the same layer in the VGG network;

[0185] The sum of the mean square error losses corresponding to each feature map in the multiple feature maps of the sample repair image is determined to obtain a content loss value of the sample repair image.

[0186] In some implementations, determining an overall loss value of the generative adversarial network based on the adversarial loss value of the sample inpainted image and the content loss value of the sample inpainted image includes:

[0187] The adversarial loss value of the sample repaired image and the content loss value of the sample repaired image are weighted summed to obtain the overall loss value of the generative adversarial network.

[0188] In some embodiments, cropping the sample repaired image includes: cropping a random number of image blocks of random sizes from random positions in the sample repaired image;

[0189] Cropping the sample real image, including: cropping a random number of image blocks of random sizes from random positions in the sample real image;

[0190] Determine the distance between each sample repaired image block and the corresponding sample real image block, including:

[0191] For a sample restored image block, determine a target sample restored image block having the same position and the same size as the sample restored image block from a plurality of sample real image blocks cropped from the sample real image;

[0192] Determine the Wasserstein distance between the sample inpainted image patch and the target sample inpainted image patch.

[0193] Figure 6 : is a structural schematic diagram of a blurred invoice image repair device 600 based on a generative adversarial network provided in an embodiment of the present application, and the device 600 includes:

[0194] An input unit 610 is used to input a blurred image to be repaired into a generator in a generative adversarial network, wherein the generator includes an attention gate module 640, an encoder 620, and a decoder 630, wherein the encoder includes a first LeWin transformer block, and the decoder includes a second LeWin transformer block and a third LeWin transformer block, wherein a jump connection is provided between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, and the attention gate module 640 is provided before the jump connection between the encoder and the decoder, and the blurred image is an invoice image;

[0195] An encoder 620, configured to encode the features of the blurred image through a first LeWin transformer block to obtain a first feature map of the blurred image;

[0196] A decoder 630, configured to decode the first feature map of the blurred image through a second LeWin transformer block to obtain a second feature map of the blurred image;

[0197] An attention gate module 640 is used to perform feature fusion on the first feature map and the second feature map to obtain a third feature map, generate an attention map based on the third feature map, and weight the third feature map based on the attention map to obtain a fourth feature map, where the attention map is used to characterize the importance of each position in the third feature map;

[0198] The decoder 630 is used to decode based on the second feature map and the fourth feature map through a third LeWin transformer block to obtain a restored image.

[0199] In some embodiments, the attention gate module 640 includes a convolution layer, an upsampling layer, and an activation function. Before the attention gate module 640 performs feature fusion on the first feature map and the second feature map, the attention gate module 640 is also used to convolve the first feature map and the second feature map respectively through the convolution layer to adjust the number of channels of the first feature map to be consistent with the number of channels of the second feature map; upsample the convolved first feature map and the convolved second feature map respectively; add the upsampled first feature map and the upsampled second feature map, and perform a nonlinear transformation on the added image through the activation function to obtain a third feature map; determine the attention score of each position in the third feature map through the convolution layer and the activation function to obtain an attention map, and the attention score is used to characterize the importance of the corresponding position.

[0200] In some embodiments, the generative adversarial network further includes a discriminator, and the device further includes: a preprocessing unit, configured to perform blurring processing on the sample real image to obtain a sample blurred image, wherein the sample real image is an invoice image;

[0201] The input unit 610 is further used to input the sample blurred image into the generator to obtain the sample repaired image; and input the sample repaired image and the sample real image into the discriminator;

[0202] A discriminator is used to crop the sample repaired image to obtain a plurality of sample repaired image blocks, and to crop the sample real image to obtain a plurality of sample real image blocks; perform feature extraction on each sample real image block to obtain the feature of each sample real image block, and perform feature extraction on each sample repaired image block to obtain the feature of each sample repaired image block; determine the distance between each sample repaired image block and the corresponding sample real image block based on the feature of each sample real image block and the feature of each sample repaired image block; perform an average operation on the distance between each sample repaired image block and the corresponding sample real image block to obtain an adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image discriminated by the discriminator and the sample real image;

[0203] The apparatus further includes: a loss determination unit, configured to determine an overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image;

[0204] The parameter adjustment unit is used to adjust the parameters of the generator and the parameters of the discriminator based on the overall loss value of the generative adversarial network.

[0205] In some embodiments, the loss determination unit is used to perform feature extraction on the sample repaired image and the sample real image respectively to obtain a feature map of the sample repaired image and a feature map of the sample real image; determine a content loss value of the sample repaired image based on the feature map of the sample repaired image and the feature map of the sample real image, and the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image; determine an overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image.

[0206] In some embodiments, the loss determination unit is used to use multiple layers of a pre-trained VGG network to perform feature extraction on the sample repair image and the sample real image respectively, so as to obtain multiple feature maps of the sample repair image output by the multiple layers and multiple feature maps of the sample real image; for the feature map of the sample repair image output by any layer of the multiple layers of the VGG network, determine the mean square error loss between the feature map of the sample repair image and the feature map of the sample real image output by the same layer in the VGG network; determine the sum of the mean square error losses corresponding to each feature map in the multiple feature maps of the sample repair image, so as to obtain the content loss value of the sample repair image.

[0207] In some embodiments, the loss determination unit is used to perform a weighted summation of the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image to obtain an overall loss value of the generative adversarial network.

[0208] In some embodiments, the discriminator is used to crop a random number of image blocks of random sizes from random positions in a sample repaired image; crop a random number of image blocks of random sizes from random positions in a sample real image; for a sample repaired image block, determine a target sample repaired image block with the same position and size as the sample repaired image block from multiple sample real image blocks cropped from the sample real image; and determine the Wasserstein distance between the sample repaired image block and the target sample repaired image block.

[0209] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0210] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0211] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects, and cannot be understood as indicating or implying relative importance. For example, the first feature map and the second feature map are used to distinguish different feature maps, rather than to describe a specific order of feature maps, and cannot be understood as the first feature map being more important than the second feature map.

[0212] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk Solid State Disk (SSD)), etc.

[0213] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for restoring blurred invoice images based on generative adversarial networks, characterized in that: The method comprises: Inputting the blurred image to be repaired into a generator in a generative adversarial network, the generator includes an attention gate module, an encoder and a decoder, the encoder includes a first LeWin transformer block, the decoder includes a second LeWin transformer block and a third LeWin transformer block, there is a jump connection between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, the attention gate module is arranged before the jump connection between the encoder and the decoder, and the blurred image is an invoice image; Encoding the features of the blurred image by using the first LeWin transformer block in the encoder to obtain a first feature map of the blurred image; Decoding the first feature map of the blurred image by the second LeWin transformer block in the decoder to obtain a second feature map of the blurred image; Performing feature fusion on the first feature map and the second feature map through the attention gate module to obtain a third feature map, generating an attention map based on the third feature map, and weighting the third feature map based on the attention map to obtain a fourth feature map, wherein the attention map is used to characterize the importance of each position in the third feature map; Decoding is performed based on the second feature map and the fourth feature map by the third LeWin transformer block in the decoder to obtain a restored image.

2. The method according to claim 1, characterized in that The attention gate module includes a convolution layer, an upsampling layer, and an activation function. Before the first feature map and the second feature map are subjected to feature fusion by the attention gate module, the method further includes: Convolving the first feature map and the second feature map respectively through the convolution layer in the attention gate module, so that the number of channels of the first feature map is adjusted to be consistent with the number of channels of the second feature map; The performing feature fusion on the first feature map and the second feature map by the attention gate module includes: Upsampling the first feature map after convolution and the second feature map after convolution respectively; The upsampled first feature map and the upsampled second feature map are added, and the added image is subjected to nonlinear transformation through an activation function to obtain a third feature map; The generating an attention map based on the third feature map comprises: The attention score of each position in the third feature map is determined by a convolution layer and an activation function to obtain an attention map, where the attention score is used to characterize the importance of the corresponding position.

3. The method according to claim 1, characterized in that The generative adversarial network further includes a discriminator. Before the blurred image to be repaired is input into the generator in the generative adversarial network, the method further includes: Performing fuzzy processing on the sample real image to obtain a sample fuzzy image, wherein the sample real image is an invoice image; Inputting the sample blurred image into the generator to obtain a sample repaired image; The sample repaired image and the sample real image are input into the discriminator, and the sample repaired image is cropped by the discriminator to obtain a plurality of sample repaired image blocks, and the sample real image is cropped to obtain a plurality of sample real image blocks; feature extraction is performed on each of the sample real image blocks to obtain the features of each of the sample real image blocks, and feature extraction is performed on each of the sample repaired image blocks to obtain the features of each of the sample repaired image blocks; based on the features of each of the sample real image blocks and the features of each of the sample repaired image blocks, the distance between each of the sample repaired image blocks and the corresponding sample real image blocks is determined; the distance between each of the sample repaired image blocks and the corresponding sample real image blocks is averaged to obtain the adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image and the sample real image discriminated by the discriminator; Determining an overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image; The parameters of the generator and the parameters of the discriminator are adjusted based on the overall loss value of the generative adversarial network.

4. The method according to claim 3, characterized in that The determining the overall loss value of the generative adversarial network based on the adversarial loss value of the sample repaired image includes: Performing feature extraction on the sample repaired image and the sample real image respectively to obtain a feature map of the sample repaired image and a feature map of the sample real image; Determine a content loss value of the sample repaired image based on the feature map of the sample repaired image and the feature map of the sample real image, wherein the content loss value is used to characterize the difference between the content of the sample repaired image and the content of the sample real image; Based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image, an overall loss value of the generative adversarial network is determined.

5. The method according to claim 4, characterized in that The extracting features of the sample repaired image and the sample real image respectively includes: Using multiple layers of a pre-trained VGG network to perform feature extraction on the sample repaired image and the sample real image respectively, so as to obtain multiple feature maps of the sample repaired image and multiple feature maps of the sample real image output by the multiple layers; The determining, based on the feature map of the sample inpainted image and the feature map of the sample real image, a content loss value of the sample inpainted image comprises: For a feature map of the sample inpainted image output by any layer of the multiple layers of the VGG network, determine a mean square error loss between the feature map of the sample inpainted image and a feature map of the sample real image output by the same layer in the VGG network; The sum of the mean square error losses corresponding to each feature map in the plurality of feature maps of the sample repaired image is determined to obtain a content loss value of the sample repaired image.

6. The method according to claim 4, characterized in that The determining, based on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image, an overall loss value of the generative adversarial network comprises: A weighted sum is performed on the adversarial loss value of the sample repaired image and the content loss value of the sample repaired image to obtain an overall loss value of the generative adversarial network.

7. The method according to claim 3, characterized in that The cropping of the sample repaired image comprises: cropping a random number of image blocks of random sizes from random positions in the sample repaired image; The cropping of the sample real image comprises: cropping a random number of image blocks of random sizes from random positions in the sample real image; The determining of the distance between each of the sample restored image blocks and the corresponding sample real image blocks comprises: For one of the sample restored image blocks, determining a target sample restored image block having the same position and the same size as the sample restored image block from a plurality of sample real image blocks cropped from the sample real image; A Wasserstein distance between the sample inpainted image block and a target sample inpainted image block is determined.

8. A device for repairing blurred invoice images based on a generative adversarial network, characterized in that: The device comprises: An input unit, used for inputting a blurred image to be repaired into a generator in a generative adversarial network, wherein the generator includes an attention gate module, an encoder and a decoder, wherein the encoder includes a first LeWin transformer block, and the decoder includes a second LeWin transformer block and a third LeWin transformer block, wherein a jump connection exists between the first LeWin transformer block of the encoder and the third LeWin transformer block of the decoder, and the attention gate module is arranged before the jump connection between the encoder and the decoder, and the blurred image is an invoice image; an encoder, configured to encode the features of the blurred image by using the first LeWin transformer block to obtain a first feature map of the blurred image; a decoder, configured to decode the first feature map of the blurred image through the second LeWin transformer block to obtain a second feature map of the blurred image; an attention gate module, configured to perform feature fusion on the first feature map and the second feature map to obtain a third feature map, generate an attention map based on the third feature map, and weight the third feature map based on the attention map to obtain a fourth feature map, wherein the attention map is used to characterize the importance of each position in the third feature map; The decoder is used to decode based on the second feature map and the fourth feature map through the third LeWin transformer block to obtain a restored image.

9. The device according to claim 8, characterized in that The attention gate module includes a convolution layer, an upsampling layer and an activation function. Before the attention gate module performs feature fusion on the first feature map and the second feature map, the attention gate module is also used to convolve the first feature map and the second feature map respectively through the convolution layer to adjust the number of channels of the first feature map to be consistent with the number of channels of the second feature map; upsample the convolved first feature map and the convolved second feature map respectively; add the upsampled first feature map and the upsampled second feature map, and perform nonlinear transformation on the added image through the activation function to obtain a third feature map; determine the attention score of each position in the third feature map through the convolution layer and the activation function to obtain an attention map, and the attention score is used to characterize the importance of the corresponding position.

10. The device according to claim 8, characterized in that The generative adversarial network further includes a discriminator, and the device further includes: a preprocessing unit, which is used to perform blurring processing on the sample real image to obtain a sample blurred image, wherein the sample real image is an invoice image; The input unit is further used to input the sample blurred image into the generator to obtain a sample repaired image; and input the sample repaired image and the sample real image into the discriminator; The discriminator is used to crop the sample repaired image to obtain a plurality of sample repaired image blocks, and to crop the sample real image to obtain a plurality of sample real image blocks; perform feature extraction on each of the sample real image blocks to obtain the features of each of the sample real image blocks, and perform feature extraction on each of the sample repaired image blocks to obtain the features of each of the sample repaired image blocks; determine the distance between each of the sample repaired image blocks and the corresponding sample real image blocks based on the features of each of the sample real image blocks and the features of each of the sample repaired image blocks; perform an average operation on the distance between each of the sample repaired image blocks and the corresponding sample real image blocks to obtain an adversarial loss value of the sample repaired image, and the adversarial loss value is used to characterize the difference between the sample repaired image and the sample real image discriminated by the discriminator; The apparatus further comprises: a loss determination unit, configured to determine an overall loss value of the generative adversarial network based on the adversarial loss value of the sample inpainted image; A parameter adjustment unit is used to adjust the parameters of the generator and the parameters of the discriminator based on the overall loss value of the generative adversarial network.

Images