Image enhancement model training and image enhancement method, device, equipment and medium
Through improved loop generation adversarial network model, using multiple U-shaped networks to train the target image sample set, the problem of poor performance of existing image enhancement models is solved, higher image enhancement accuracy and effect are achieved, and hardware resources are reduced need.
Patent Information
- Application Number
- CN202210151826.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-02-18
AI Technical Summary
The image enhancement effect of existing image enhancement models needs to be improved, and traditional methods are difficult to cover all situations in full, and machine learning methods such as unsupervised learning require high-cost data annotation or are not effective.
By obtaining the target image sample set, including images with low contrast and high contrast, trained using an improved loop generation adversarial network model, which contains multiple U-shaped networks, trains parameters by predicting the loss function formed by the image and the target image until the result is Target image enhancement model.
It improves image enhancement accuracy and effect, and is suitable for document photo enhancement processing scenarios with high accuracy requirements, reducing the consumption of hardware resources.
Smart Images

Figure CN114529469B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image enhancement model training and image enhancement method, device, equipment and medium. Background Art
[0002] In real life, we often need to scan documents. There are many applications that can scan documents on the market. The core step of document scanning is to obtain a high-contrast grayscale image through image enhancement. After the image enhancement process, the noise is removed, but some information in the image will also be lost.
[0003] At present, common image enhancement methods are mainly divided into traditional methods and machine learning methods. The main disadvantage of traditional methods is that it is difficult to fully cover various situations, and the actual effect depends significantly on the change of parameters. Existing machine learning methods include supervised machine learning methods and unsupervised learning methods. Supervised learning methods require manual annotation of a large amount of paired training data to achieve good results, while manual annotation of image sharpening data is difficult and requires high costs. Unsupervised learning methods usually use adversarial generative networks, and the image enhancement effect needs to be improved. Summary of the invention
[0004] The present invention provides an image enhancement model training and image enhancement method, device, equipment and medium to solve the problem that the image enhancement effect of the traditional image enhancement model needs to be improved, and improves the traditional adversarial generation network to improve the image enhancement accuracy and enhancement effect.
[0005] According to one aspect of the present invention, a method for training an image enhancement model is provided, comprising:
[0006] Acquire a target image sample set; the target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target images is lower than the contrast of the second target images;
[0007] Input the target image in the target image sample set into a cyclic generative adversarial network model to obtain a predicted image; wherein the generative network of the cyclic generative adversarial network model includes: a plurality of U-shaped networks;
[0008] Training parameters of the cyclic generative adversarial network model according to a loss function formed by the predicted image and the target image;
[0009] Return to executing the operation of inputting the target image in the target image sample set into the recurrent generative adversarial network model to obtain a predicted image until a target image enhancement model is obtained.
[0010] According to another aspect of the present invention, there is provided an image enhancement method, comprising:
[0011] Get the image to be processed;
[0012] The image to be processed is subjected to image enhancement processing by using a target image enhancement model to obtain an enhanced image.
[0013] According to another aspect of the present invention, there is provided a training device for an image enhancement model, comprising:
[0014] An acquisition module is used to acquire a target image sample set; the target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target image is lower than the contrast of the second target image;
[0015] A prediction module, used for inputting the target image in the target image sample set into a recurrent generative adversarial network model to obtain a predicted image; wherein the generative network of the recurrent generative adversarial network model includes: a plurality of U-shaped networks;
[0016] A training module, used for training the parameters of the recurrent generative adversarial network model according to the loss function formed by the predicted image and the target image;
[0017] The iterative module is used to return to the operation of inputting the target image in the target image sample set into the cyclic generative adversarial network model to obtain a predicted image until a target image enhancement model is obtained.
[0018] According to another aspect of the present invention, there is provided an image enhancement device, comprising:
[0019] An acquisition module, used for acquiring an image to be processed;
[0020] The processing module is used to perform image enhancement processing on the image to be processed through a target image enhancement model to obtain an enhanced image.
[0021] According to another aspect of the present invention, a computer device is provided, the computer device comprising:
[0022] at least one processor; and
[0023] a memory communicatively connected to the at least one processor; wherein,
[0024] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image enhancement model training method or image enhancement method described in any embodiment of the present invention.
[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image enhancement model training method or image enhancement method described in any embodiment of the present invention when executed.
[0026] The technical solution of the embodiment of the present invention is as follows: obtaining a target image sample set; the target image sample set includes: multiple first target images and multiple second target images; the contrast of the first target image is lower than the contrast of the second target image; inputting the target image in the target image sample set into a cyclic generative adversarial network model to obtain a predicted image; the generative network of the cyclic generative adversarial network model includes: multiple U-shaped networks; training the parameters of the cyclic generative adversarial network model according to the loss function formed by the predicted image and the target image; returning to execute the operation of inputting the target image in the target image sample set into the cyclic generative adversarial network model to obtain the predicted image, until the target image enhancement model is obtained, thereby solving the problem that the image enhancement effect of the traditional image enhancement model needs to be improved, improving the traditional generative adversarial network, and achieving the technical effect of improving the image enhancement accuracy and enhancement effect.
[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0029] Figure 1 is a flowchart of a training method for an image enhancement model provided according to Embodiment 1 of the present invention;
[0030] Figure 2 It is a schematic diagram of the structure of the cyclic generative adversarial network;
[0031] Figure 3 A generative network of a cyclic generative adversarial network model provided by Embodiment 1 of the present invention;
[0032] Figure 4 This is a schematic diagram of the network structure of the traditional U-net network;
[0033] Figure 5 This is a schematic diagram of a network structure of a U-shaped network provided according to the first embodiment of the present invention;
[0034] Figure 6 is a schematic diagram of the structure of a dilated convolution block provided according to the first embodiment of the present invention;
[0035] Figure 7 This is a schematic diagram of the structure of an upsampling block in a generation network provided by Embodiment 1 of the present invention;
[0036] Figure 8 is a schematic diagram of the structure of a downsampling block in a generation network provided by Embodiment 1 of the present invention;
[0037] Fig. 9 is a flow chart of an image enhancement method provided according to Embodiment 2 of the present invention;
[0038] Fig.10 is a structural schematic diagram of a training device for an image enhancement model provided according to Embodiment 3 of the present invention;
[0039] Fig.11 is a structural schematic diagram of an image enhancement device provided according to a fourth embodiment of the present invention;
[0040] Fig.12 It is a structural schematic diagram of a computer device for implementing the training method of the image enhancement model of an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0043] Embodiment 1
[0044] Figure 1 A flowchart of a method for training an image enhancement model is provided for the first embodiment of the present invention. This embodiment is applicable to the case of training an image enhancement model. The method can be performed by a training device for an image enhancement model. The training device for an image enhancement model can be implemented in the form of hardware and / or software. The training device for an image enhancement model can be configured in a computer device. Figure 1 As shown, the method includes:
[0045] S110, obtaining a target image sample set; the target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target image is lower than the contrast of the second target image.
[0046] The target image sample set is a training sample set for training an image enhancement model, and the target image sample set may include multiple first target images and multiple second target images, and the contrast of the first target image is lower than the contrast of the second target image. Specifically, the first target image may be a low-contrast image, and the second target image may be a high-contrast image. It should be noted that the first target image and the second target image do not need to be associated with each other.
[0047] Exemplarily, the first target image may be obtained by scanning a paper document with a scanning device, or by photographing a paper document with a computer device having an image acquisition function, or by filtering or intercepting the photographed image to obtain the first target image. The second target image may be obtained by converting an electronic document in PDF or Word format corresponding to the paper document into a picture format such as PNG with a fixed resolution size. The embodiment of the present invention does not limit the method for obtaining a sample set of target images. It should be noted that the first target image has a low contrast, but generally has a high resolution.
[0048] S120, inputting the target image in the target image sample set into a recurrent generative adversarial network model to obtain a predicted image; wherein the generative network of the recurrent generative adversarial network model includes: multiple U-shaped networks.
[0049] Among them, CycleGan is an unsupervised learning model derived from Generative Adversarial Network (GAN). Generative Adversarial Network generally consists of a generative network and a discriminative network. During the training process, the generative network is responsible for generating predicted data, making it difficult to distinguish between predicted data and real data, while the discriminative network strives to distinguish between predicted data and real data. During the training process, the two are constantly in confrontation. Figure 2As shown in the figure, the cyclic generative adversarial network uses two generating networks G and F, and two discriminating networks Dx and Dy. The first generating network G represents the generating network from domain X to domain Y, and the second generating network F represents the generating network from domain Y to domain X. The first discriminating network D X The second identification network D is used to determine whether F(y) generated by the first generation network F belongs to the X domain. Y Then determine whether G(x) generated by the second generation network G belongs to the Y domain. X and Y represent two different domains, the X domain can be represented as a sample image domain, and the Y domain can be represented as a predicted image domain. The image output by the first generation network can be used as an input image of the second generation network, and the output image of the second generation network should match the original input image. Exemplarily, the X domain can represent a first target image, and the Y domain can represent a second target image.
[0050] In the cyclic generative adversarial network model, the two generative networks have the same structure. In the traditional cyclic generative adversarial network model, the commonly used generative adversarial network is a U-net network. U-net uses a fully convolutional neural network. The left network is a feature extraction network composed of multiple downsampling blocks, and the right network is a feature fusion network composed of multiple upsampling blocks.
[0051] Traditional cyclic generative adversarial network models are generally used for processing lower resolution images. For processing higher resolution images, the two generative networks of the cyclic generative adversarial network model in the embodiment of the present invention respectively include multiple U-shaped networks, and multiple U-shaped networks are connected as a generative network to improve the accuracy of the generative model.
[0052] Exemplarily, the method of interconnecting multiple U-shaped networks can be to connect two adjacent U-shaped networks, and the specific connection method is that the input end and the output end of the current U-shaped network are respectively connected to the input end of the next U-shaped network, and the output end of each U-shaped network is respectively used as an output end of the generated network. Therefore, the generated network has multiple output ends, and the number of output ends is the same as the number of U-shaped networks included in the generated network. The number of U-shaped networks can be determined according to the requirements of the actual application scenario for accuracy, and the embodiment of the present invention does not impose any restrictions on this.
[0053] Specifically, a target image in a target image sample set is input into a cyclic generative adversarial network model to obtain a predicted image, and the input target image may be a first target image or a second target image.
[0054] S130, generating parameters of the adversarial network model according to the loss function training loop formed by the predicted image and the target image.
[0055] Specifically, the predicted image obtained by inputting the first target image into the cyclic generative adversarial network model is matched with the first target image to form a loss function between the two, and the parameters of each generative network in the cyclic generative adversarial network model are trained. This is the forward training cycle process of the cyclic consistency adversarial network from the X domain to the Y domain. Similarly, in the reverse training cycle process from the Y domain to the X domain, the reverse is reversed, and the predicted image obtained by inputting the second target image into the cyclic generative adversarial network model is matched with the second target image to form a loss function between the two, and the parameters of each generative network in the cyclic generative adversarial network model are trained.
[0056] Exemplarily, the loss function formed by the predicted image and the target image may include: an adversarial loss consisting of a first adversarial loss corresponding to the first generation network and a second adversarial loss corresponding to the second generation network, wherein the first adversarial loss L GAN(G,DY,X,Y) for:
[0057] L GAN(G,DY,X,Y) =E y~pdata(y) [log D Y (y)]+E x~pdata(x) [log(1-D Y (G(x))];
[0058] The second adversarial loss is:
[0059] L GAN(F,DX,Y,X) =E x~pdata(x) [log D X (x)]+E y~pdata(y) [log(1-D X (F(y))];
[0060] Among them, D X represents the first discriminant network and D Y represents the second identification network, G represents the first generation network, and F represents the second generation network.
[0061] Since the first generation network G and the second generation network F can easily deceive the identification network together in the unsupervised training of the cycle-generated adversarial network model, training CycleGAN using only the GAN loss cannot ensure the consistency of the cycle. An additional cycle consistency loss is required to enforce this property. The ring consistency loss is defined as the difference between the input value x and the forward prediction F(G(x)) and the input value y and the forward prediction G(F(y)). The larger the difference, the farther the predicted image is from the target image. Ideally, our network minimizes this loss. The cycle consistency loss is:
[0062] L cyc(G,F) =E x~pdata(x) [||F(G(x))-x||1]+E y~pdata(y)[||G(F(y))-y||1];
[0063] Among them, L cyc(G,F) is the cycle consistency loss; E x~pdata(x) is the difference between the input value x and the forward prediction F(G(x)), E y~pdata(y) is the difference between the input value y and the forward prediction G(F(y)). The cycle-consistent loss can be an L1 norm loss function, i.e., a mean absolute error function; or it can be a picture structural similarity function (structural similarity, SSIM), or a multi-scale picture structural similarity function (multi-scale structural similarity, MS-SSIM); or it can be a weighted sum of multiple functions.
[0064] In summary, the loss function formed by the predicted image and the target image is:
[0065] L (G,F,DX,DY) =L GAN(G,DY,X,Y) +L GAN(F,DX,Y,X) +λL cyc(G,F) ;
[0066] Among them, L (G,F,DX,DY) is the loss function, L GAN(G DY,X,Y) is the first adversarial loss, L GAN(F,DX,Y,X) is the second adversarial loss; λ is the weight of the cycle-consistent loss.
[0067] S140, returning to execute the operation of inputting the target image into the cyclic generative adversarial network model to obtain a predicted image, until the target image enhancement model is obtained.
[0068] Specifically, the target images in the target image sample set are iteratively input into the recurrent generative adversarial network model in turn to obtain predicted images, and the parameters of the recurrent generative adversarial network model are trained according to the loss function formed by the predicted images and the target images until the loss function converges or reaches a preset number of training times, and the recurrent generative adversarial network model at this time is used as the target image enhancement model for image enhancement.
[0069] The training method of the image enhancement model provided by the embodiment of the present invention is through obtaining a target image sample set; the target image sample set includes: multiple first target images and multiple second target images; the contrast of the first target image is lower than the contrast of the second target image; the target image in the target image sample set is input into a cyclic generative adversarial network model to obtain a predicted image; wherein the generative network of the cyclic generative adversarial network model includes: multiple U-shaped networks; the parameters of the cyclic generative adversarial network model are trained according to the loss function formed by the predicted image and the target image; return to execute the operation of inputting the target image in the target image sample set into the cyclic generative adversarial network model to obtain the predicted image, until the target image enhancement model is obtained, the adversarial generation network of the traditional cyclic generative adversarial network model is improved to obtain a target image enhancement model including multiple U-shaped networks, which can improve the image enhancement accuracy and enhancement effect.
[0070] On the basis of the above embodiment, the structure of the cyclic generative adversarial network model is further refined. It should be noted that, in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.
[0071] In one embodiment, the cyclic generative adversarial network model includes: a first generative network from a sample image domain to a predicted image domain and a second generative network from a predicted image domain to a sample image domain; wherein the first generative network and the second generative network are respectively composed of a first U-shaped network and a second U-shaped network, and the input end of the first U-shaped network and the output end of the first U-shaped network are respectively connected to the input end of the second U-shaped network; the first U-shaped network and the second U-shaped network have the same network structure.
[0072] Among them, in Figure 2 Based on the structure of the cyclic generative adversarial network model shown, in this embodiment, the first generative network G and the second generative network F are both connected by two U-shaped networks with the same network structure.
[0073] Specifically, Figure 3 As shown, the two identical U-shaped networks are the first U-shaped network and the second U-shaped network, the input end of the first U-shaped network and the output end of the first U-shaped network are respectively connected to the input end of the second U-shaped network, and the input and output of the first U-shaped network serve together as the input of the second U-shaped network.
[0074] When outputting the predicted image, the output image of the second U-type network is used as the predicted image, and the output result of the first U-type network is only used as the prior result. In the parameter training process of the cyclic generative adversarial network model, the output of the first U-type network and the output of the second U-type network are used together to calculate the loss function value to optimize the generated network.
[0075] This embodiment improves the image enhancement accuracy of the recurrent generative adversarial network model by connecting two U-shaped networks as the generative network of the recurrent generative adversarial network model. It is suitable for document photo enhancement processing scenarios with extremely high precision requirements, so that the captured pictures are clear and visible.
[0076] In one embodiment, the network structure of the U-type network includes: a first preset number of continuous upsampling blocks and a first preset number of continuous downsampling blocks; wherein the continuous upsampling blocks and the continuous downsampling blocks are connected by hole convolution blocks, and the hole convolution blocks include: a second preset number of hole convolutions.
[0077] For example, Figure 4 It is the network structure of the traditional U-Net. For a 256×256×3 image, a deep sampling process of 7 downsampling and 7 upsampling is required to make the output of the U-Net network cover a larger field of view. Furthermore, using two U-Nets as the generative network of the recurrent generative adversarial network model will cause the network to be too deep, which will greatly increase the hardware requirements for photos of the same size. Therefore, in order to reduce the occupation of hardware resources, we added a hole convolution block between the downsampling convolution process and the upsampling convolution process so that a single convolution can obtain a large field of view, thereby replacing the deep sampling in the traditional U-Net network to ensure the field of view while reducing the number of samplings to reduce the consumption of hardware resources.
[0078] Specifically, Figure 5 As shown, the network structure of the U-shaped network includes: a first preset number of continuous upsampling blocks and a first preset number of continuous downsampling blocks; wherein the continuous upsampling blocks and the continuous downsampling blocks are connected by dilated convolution blocks. The dilated convolution blocks are formed by splicing a second preset number of dilated convolutions.
[0079] During the forward propagation process, the output of the downsampling block is used as the input result of the first dilated convolution. During the operation of the dilated convolution block, the input result and output result of each dilated convolution are used as the input result of a splicing. The output of each splicing is used as the input result of the next dilated convolution until the output result of the last splicing is used as the output result of the dilated convolution block, and the output result is input into the continuous upsampling block.
[0080] It should be noted that the first preset number of upsampling blocks and downsampling blocks and the second preset number of dilated convolutions in the dilated convolution block are interconnected, and the second preset number can be determined by the size of the image after the image is upsampled by the first preset number of consecutive upsampling blocks. The first preset number and the second preset number can be set according to actual needs, and the embodiment of the present invention does not limit this.
[0081] For example, Figure 6 It is a schematic diagram of the structure of the hole convolution block, such as Figure 6 As shown in the figure, the dilated convolution block is composed of 4 dilated convolutions. The step size of each dilated convolution is 1, the convolution kernel is 3, and the distance between two adjacent values during the dilated convolution operation is set to the coefficient L. When L is 1, it is the same as the ordinary convolution. Figure 6 The coefficients of the four dilated convolutions in the dilated convolution block shown are 1, 2, 3, and 5 respectively.
[0082] This embodiment uses a hole convolution block connection between continuous upsampling blocks and continuous downsampling blocks to replace the deep sampling in the traditional U-net network to ensure the field of view while reducing the number of samplings to reduce the consumption of hardware resources.
[0083] The images generated by the recurrent generative adversarial network model formed by the generative network provided in the above embodiments often have a checkerboard-like shadow pattern, which is particularly prominent in the process of document photo image enhancement, especially in the generated clear images where the fonts are dense, that is, the color brightness of the image changes frequently and drastically. This problem also exists in the U-Net generative network used in the traditional recurrent generative adversarial network model.
[0084] In response to the above problem, in one embodiment, the upsampling block includes: a first activation unit, a bilinear interpolation unit, a convolution unit and a first normalization unit; the downsampling block includes: a second activation unit, a hole convolution unit, a maximum pooling unit and a second normalization unit.
[0085] Figure 7 It is a structural diagram of the upsampling block in the generative network provided by an embodiment of the present invention. The pictures generated by the cyclic generative adversarial network model often have a checkerboard-like shadow pattern due to the transposed convolution (deconvolution) used in the upsampling block part of the generative network. When the stride and size in the transposed convolution are not completely matched, overlapping parts will appear, resulting in shadows in the result. Reasonable selection of the stride and size in the transposed convolution requires complex testing and training. Therefore, the embodiment of the present invention improves the upsampling block. In the upsampling block, the bilinear interpolation unit is first used to upsample the input target image to double its length and width, and then ordinary convolution is used to replace the original transposed convolution operation. Bilinear interpolation is a commonly used interpolation method when changing size in image processing. The present invention does not elaborate on the bilinear interpolation method.
[0086] Specifically, Figure 7As shown, in the embodiment of the present invention, the upsampling block includes: a first activation unit, a bilinear interpolation unit, a convolution unit and a first normalization unit. The activation function in the first activation unit can be, for example, a ReLU function, and the convolution kernel of the convolution unit is 5×5 and the step size is 1.
[0087] Accordingly, the downsampling block is also adjusted to adapt to the improved upsampling block. Figure 8 As shown, the downsampling block specifically includes a second activation unit, a hole convolution unit, a maximum pooling unit, and a second normalization unit. The second activation unit can be, for example, a LeakyReLU activation function, which is a common improvement of ReLU. A small gradient is also added in the interval less than 0, which can prevent the necrosis of network nodes to a certain extent. The normalization unit is a common layer following the convolution, which can generally accelerate the convergence speed of the network and improve the convergence stability.
[0088] In one embodiment, the loss function includes: a first adversarial loss corresponding to the first generative network, a second adversarial loss corresponding to the second generative network, and a cycle-consistent loss between the first generative network and the second generative network; wherein the cycle-consistent loss includes a weighted sum of a multi-scale image structure similarity function and a mean absolute error function with a first preset weight.
[0089] Specifically, the image enhancement process is not a symmetrical mutual conversion process. In the image enhancement process, that is, in the process of the first generator G generating an image, some information will undoubtedly be lost. What we expect is to be able to discard all noise, including sensor noise during image acquisition, block effects caused by lossy compression during image storage, and other noise. However, in the process of the second generator F generating an image, due to the existence of cyclic loss, the second generator F will try its best to generate a photo close to the original image, that is, it generates an image containing various noises. This is undoubtedly a great interference to the entire model.
[0090] To this end, the cycle-consistent loss includes a weighted sum of a multi-scale image structure similarity function and a mean absolute error function with a first preset weight. The multi-scale image structure similarity function is an image quality evaluation index based on the assumption that the human eye extracts structural information from an image when viewing it. Compared with the mean absolute error function (L1 loss function), the multi-scale image structure similarity function can better reflect the degree of image similarity in the eyes of humans.
[0091] In one embodiment, acquiring a target image sample set includes:
[0092] Access to paper and electronic documents;
[0093] Capturing the image of the paper document by an image acquisition device to obtain a first target image in a target image sample set;
[0094] The electronic document is converted into a picture format to obtain a second target image in the target image sample set.
[0095] The image acquisition device may be a device with image acquisition function such as a scanning device and a camera, and the image acquisition device may also have certain image pre-processing functions such as filtering.
[0096] Specifically, a document photo of a paper document is captured by an image capture device, and the document photo is used as the first target image in the target image sample set; an electronic document in a format such as word or pdf is converted into a picture format such as JPEG or PNG with a fixed resolution size as the second target image in the target image sample set.
[0097] In order to reduce interference to model training, the first target image may be cropped or other preprocessed before being input into the model, so that the first target image has a fixed size, such as 256×256, and parts other than the document do not appear in the first target image as much as possible.
[0098] Embodiment 2
[0099] Fig. 9 A flowchart of an image enhancement method is provided for the second embodiment of the present invention. This embodiment is applicable to the case of performing image enhancement on a document image. The method can be performed by an image enhancement device. The image enhancement device can be implemented in the form of hardware and / or software. The image enhancement device can be configured in a computer device. Fig. 9 As shown, the method includes:
[0100] S210: Obtain an image to be processed.
[0101] Specifically, the method for obtaining the image to be processed may be to capture the image to be processed by a camera, or to obtain the image to be processed by interception. The embodiment of the present invention does not limit the method for obtaining the image to be processed.
[0102] S220: Perform image enhancement processing on the image to be processed by using a target image enhancement model to obtain an enhanced image.
[0103] Among them, the target image enhancement model is obtained by iteratively training the cyclic generative adversarial network model with the target image sample set. The generative network of the generative adversarial network model includes: multiple U-shaped networks. In the cyclic generative adversarial network model, the two generative networks have the same structure. In the traditional cyclic generative adversarial network model, the commonly used generative adversarial network is a U-shaped network U-Net. U-Net adopts a fully convolutional neural network. The left network is a feature extraction network composed of multiple downsampling blocks, and the right network is a feature fusion network composed of multiple upsampling blocks.
[0104] Traditional cyclic generative adversarial network models are generally used for processing lower resolution images. For processing higher resolution images, the two generative networks of the cyclic generative adversarial network model in the embodiment of the present invention respectively include multiple U-shaped networks, and multiple U-shaped networks are connected as a generative network to improve the accuracy of the generative model.
[0105] The image enhancement method provided in the embodiment of the present invention obtains an image to be processed; performs image enhancement processing on the image to be processed through a target image enhancement model to obtain an enhanced image; wherein the adversarial generative network in the traditional image enhancement model is improved to obtain a target image enhancement model including multiple U-shaped networks, which can improve the image enhancement accuracy and enhancement effect.
[0106] Embodiment 3
[0107] Fig.10 This is a schematic diagram of the structure of a training device for an image enhancement model provided in Embodiment 3 of the present invention. Fig.10 As shown, the device includes: an acquisition module 310, a prediction module 320, a training module 330 and an iteration module 340;
[0108] The acquisition module 310 is used to acquire a target image sample set; the target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target image is lower than the contrast of the second target image;
[0109] The prediction module 320 is used to input the target image in the target image sample set into a recurrent generative adversarial network model to obtain a predicted image; wherein the generative network of the recurrent generative adversarial network model includes: a plurality of U-shaped networks;
[0110] A training module 330, configured to train parameters of the recurrent generative adversarial network model according to a loss function formed by the predicted image and the target image;
[0111] The iteration module 340 is used to return to the operation of inputting the target image in the target image sample set into the recurrent generative adversarial network model to obtain a predicted image until a target image enhancement model is obtained.
[0112] Optionally, the cyclic generative adversarial network model includes: a first generative network from a sample image domain to a predicted image domain and a second generative network from a predicted image domain to a sample image domain;
[0113] Among them, the first generation network and the second generation network are respectively composed of a first U-shaped network and a second U-shaped network, the input end of the first U-shaped network and the output end of the first U-shaped network are respectively connected to the input end of the second U-shaped network; the first U-shaped network and the second U-shaped network have the same network structure.
[0114] Optionally, the network structure includes: a first preset number of consecutive upsampling blocks and a first preset number of consecutive downsampling blocks;
[0115] The continuous upsampling blocks and the continuous downsampling blocks are connected by using a dilated convolution block, and the dilated convolution block includes: a second preset number of dilated convolutions.
[0116] Optionally, the upsampling block includes: a first activation unit, a bilinear interpolation unit, a convolution unit and a first normalization unit; the downsampling block includes: a second activation unit, a hole convolution unit, a maximum pooling unit and a second normalization unit.
[0117] Optionally, the loss function includes: a first adversarial loss corresponding to the first generative network, a second adversarial loss corresponding to the second generative network, and a cycle-consistent loss between the first generative network and the second generative network;
[0118] The cycle-consistent loss includes a weighted sum of a multi-scale image structure similarity function and a mean absolute error function with a first preset weight.
[0119] Optionally, the acquisition module 310 is specifically used for:
[0120] Access to paper and electronic documents;
[0121] Capturing the image of the paper document by an image acquisition device to obtain a first target image in the target image sample set;
[0122] The electronic document is converted into a picture format to obtain a second target image in the target image sample set.
[0123] The image enhancement model training device provided in the embodiment of the present invention can execute the image enhancement model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0124] Embodiment 4
[0125] Fig.11This is a schematic diagram of the structure of an image enhancement device provided by Embodiment 4 of the present invention. Fig.11 As shown, the device includes: an acquisition module 410 and a processing module 420;
[0126] Wherein, the acquisition module 410 is used to acquire the image to be processed;
[0127] The processing module 420 is used to perform image enhancement processing on the image to be processed through a target image enhancement model to obtain an enhanced image.
[0128] The image enhancement device provided in the embodiment of the present invention can execute the image enhancement method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0129] Embodiment 5
[0130] Fig.12 A schematic diagram of a computer device 10 that can be used to implement an embodiment of the present invention is shown. The computer device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The computer device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0131] like Fig.12 As shown, the computer device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the computer device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0132] A number of components in the computer device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the computer device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0133] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a training method for an image enhancement model or an image enhancement method.
[0134] In some embodiments, the training method of the image enhancement model or the image enhancement method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the computer device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the image enhancement model or the image enhancement method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the image enhancement model or the image enhancement method in any other appropriate manner (e.g., by means of firmware).
[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0136] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0137] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0139] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0140] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0142] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A training method for an image enhancement model, It is characterized in that include: Obtain a target image sample set; The target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target images is lower than the contrast of the second target images; Input the target image in the target image sample set into a cyclic generative adversarial network model to obtain a predicted image; wherein the generative network of the cyclic generative adversarial network model includes: a plurality of U-shaped networks, and the cyclic generative adversarial network model includes: a first generative network from a sample image domain to a predicted image domain and a second generative network from a predicted image domain to a sample image domain; The first generation network and the second generation network are respectively composed of a first U-shaped network and a second U-shaped network, the input end of the first U-shaped network and the output end of the first U-shaped network are respectively connected to the input end of the second U-shaped network; the first U-shaped network and the second U-shaped network have the same network structure; Training parameters of the cyclic generative adversarial network model according to a loss function formed by the predicted image and the target image; Return to executing the operation of inputting the target image in the target image sample set into the recurrent generative adversarial network model to obtain a predicted image until a target image enhancement model is obtained.
2. The method according to claim 1, It is characterized in that The network structure includes: a first preset number of consecutive upsampling blocks and a first preset number of consecutive downsampling blocks; The continuous upsampling blocks and the continuous downsampling blocks are connected by using a dilated convolution block, and the dilated convolution block includes: a second preset number of dilated convolutions.
3. The method according to claim 2, It is characterized in that The up-sampling block includes: a first activation unit, a bilinear interpolation unit, a convolution unit and a first normalization unit; the down-sampling block includes: a second activation unit, a hole convolution unit, a maximum pooling unit and a second normalization unit.
4. The method according to any one of claims 1 to 3, It is characterized in that The loss function includes: a first adversarial loss corresponding to the first generating network, a second adversarial loss corresponding to the second generating network, and a cycle-consistent loss between the first generating network and the second generating network; The cycle-consistent loss includes a weighted sum of a multi-scale image structure similarity function and a mean absolute error function with a first preset weight.
5. An image enhancement method, It is characterized in that include: Get the image to be processed; The image to be processed is subjected to image enhancement processing by the target image enhancement model described in any one of claims 1 to 4 to obtain an enhanced image.
6. A training device for an image enhancement model, It is characterized in that include: An acquisition module, used for acquiring a target image sample set; The target image sample set includes: a plurality of first target images and a plurality of second target images; the contrast of the first target images is lower than the contrast of the second target images; A prediction module, used for inputting the target image in the target image sample set into a cyclic generative adversarial network model to obtain a predicted image; wherein the generative network of the cyclic generative adversarial network model comprises: a plurality of U-shaped networks, and the cyclic generative adversarial network model comprises: a first generative network from a sample image domain to a predicted image domain and a second generative network from a predicted image domain to a sample image domain; The first generation network and the second generation network are respectively composed of a first U-shaped network and a second U-shaped network, the input end of the first U-shaped network and the output end of the first U-shaped network are respectively connected to the input end of the second U-shaped network; the first U-shaped network and the second U-shaped network have the same network structure; A training module, used for training the parameters of the recurrent generative adversarial network model according to the loss function formed by the predicted image and the target image; The iterative module is used to return to the operation of inputting the target image in the target image sample set into the cyclic generative adversarial network model to obtain a predicted image until a target image enhancement model is obtained.
7. An image enhancement device, It is characterized in that include: An acquisition module, used for acquiring an image to be processed; A processing module, used to perform image enhancement processing on the image to be processed by using the target image enhancement model described in any one of claims 1-4 to obtain an enhanced image.
8. A computer device, It is characterized in that The computer device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the image enhancement model described in any one of claims 1-4 or execute the image enhancement method described in claim 5.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the image enhancement model described in any one of claims 1 to 4 or execute the image enhancement method described in claim 5 when executed.
