A method, system, apparatus and medium for enhancing the image of photographed documents.
By employing a dual deep convolutional neural network architecture, combined with shadow mapping and illumination correction techniques, the problem of image degradation in complex photographed documents was solved, achieving efficient image enhancement and improving imaging quality and the performance of optical character recognition systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-06-15
- Publication Date
- 2026-07-03
AI Technical Summary
Existing deep learning enhancement methods cannot effectively handle complex image degradation problems in photographed documents, such as shadows caused by objects blocking the light source, shadows caused by uneven light sources, shadows caused by uneven paper, low contrast caused by insufficient light sources, and text penetration, which affect image quality and the performance of optical character recognition systems.
A dual-depth convolutional neural network architecture is adopted. First, the shadow map is obtained and the illumination is corrected through the first deep convolutional neural network. Then, the document image and the shadow map are concatenated in the channel dimension and input into the second deep convolutional neural network for enhancement processing. The Lambertian reflection model and multi-scale output and multi-loss function supervision strategy are used for training.
It can effectively handle various light degradations and detail noises, improve the quality of photographed document images, and enhance the performance of optical character recognition systems.
Smart Images
Figure CN116823650B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of pattern recognition and artificial intelligence, and in particular to a method, system, apparatus and medium for enhancing photographic documents. Background Technology
[0002] With the development of semiconductor technology, the built-in cameras of mobile devices are becoming increasingly advanced, and their image quality is improving. Using built-in cameras to take photos and digitize document images has become a convenient method of document digitization. However, due to issues such as uneven lighting, insufficient lighting, light source obstruction, and uneven paper, the resulting document images often suffer from shadow / light degradation. Furthermore, text bleeding and various imaging noises further degrade the quality of the imaged document. These problems not only affect the aesthetics and readability of the imaged document but also impact the performance of optical character recognition systems. Deep learning-based document image enhancement methods have achieved significant performance improvements compared to traditional methods. However, existing deep learning enhancement methods are often limited by training data and method design, and can only handle single degradations, such as removing shadows caused by light source obstruction. They cannot meet the needs of document images with complex degradations encountered in real-world applications. Summary of the Invention
[0003] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, system, device and medium for enhancing the image of photographed documents.
[0004] The technical solution adopted in this invention is:
[0005] A method for enhancing photographed document images includes the following steps:
[0006] Obtain the first document image and the shadow image corresponding to the first document image;
[0007] The second document image is obtained by performing illumination correction processing based on the first document image and the obtained shadow map;
[0008] The first document image and the second document image are concatenated along the channel dimension and then input into a preset second deep convolutional neural network for enhancement processing, outputting a third document image as the final enhancement result.
[0009] Further, obtaining the shadow map corresponding to the first document image includes:
[0010] The first document image is scaled to a preset resolution and input into a preset first deep convolutional neural network to output a shadow map corresponding to the first document image.
[0011] The resolution of the shadow image is scaled to match the resolution of the first document image.
[0012] Furthermore, the first deep convolutional neural network is trained in the following manner:
[0013] Acquire multiple photographic document images, perform enhancement processing on the photographic document images, and obtain an enhanced dataset;
[0014] Based on the Lambertian reflection model, the images in the augmented dataset are processed to obtain the corresponding shadow maps, which serve as labels for model training; a training set is constructed based on the augmented dataset and the obtained shadow maps.
[0015] The first deep convolutional neural network was constructed using the UNeXt segmentation model as the network structure.
[0016] The first deep convolutional neural network is trained using the training set to obtain the trained first deep convolutional neural network;
[0017] The loss function used during training is L1 loss.
[0018] Further, the step of performing illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image includes:
[0019] Perform a pixel-level division operation between the first document image and the obtained shadow image to obtain the second document image after illumination correction.
[0020] Furthermore, the second document image is obtained by processing it using the following formula:
[0021] R = I / S
[0022] Where I is the first document image and S is the shadow image.
[0023] Furthermore, the training phase of the second deep convolutional neural network employs a strategy of multi-scale output and multi-loss function supervision.
[0024] Among them, the multi-scale output strategy uses multiple convolutional output layers, takes the intermediate layer features as input, and predicts multiple outputs with different resolutions. The additional convolutional output layers are only needed in the training stage and are discarded in the forward testing stage to obtain higher efficiency.
[0025] The multi-loss function strategy designs different loss functions for outputs at different resolutions to supervise and achieve better results.
[0026] Furthermore, the second deep convolutional neural network is obtained by training in the following manner:
[0027] Multiple photographed document images are acquired, and the photographed document images are enhanced using preset image processing software to obtain enhanced document images as labels for model training; a training set is constructed based on the photographed document images and labels;
[0028] The second deep convolutional neural network is constructed using the UNeXt segmentation model as the network structure; wherein the input of the second deep convolutional neural network consists of the photographed document image and the shadow map corresponding to the photographed document image;
[0029] The constructed second deep convolutional neural network is trained using the training set to obtain the trained second deep convolutional neural network;
[0030] Among them, in order to obtain multi-scale output In UNeXt, an output layer is connected after each skip connection. After the model training is complete... The output layer can be removed, leaving only the output layer. The output layer.
[0031] Another technical solution adopted in this invention is:
[0032] An image enhancement system for photographed documents, comprising:
[0033] An image acquisition module is used to acquire a first document image and to acquire a shadow image corresponding to the first document image;
[0034] The image correction module is used to perform illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image;
[0035] The enhancement processing module is used to concatenate the first document image and the second document image along the channel dimension, input them into a preset second deep convolutional neural network for enhancement processing, and output a third document image as the final enhancement result.
[0036] Another technical solution adopted in this invention is:
[0037] An image enhancement device for photographed documents, comprising:
[0038] At least one processor;
[0039] At least one memory for storing at least one program;
[0040] When the at least one program is executed by the at least one processor, the at least one processor performs the method as described above.
[0041] Another technical solution adopted in this invention is:
[0042] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.
[0043] The beneficial effects of this invention are: it can handle various types of lighting degradation, including shadows caused by objects blocking the light source, shadows caused by uneven light sources, shadows caused by uneven paper, and low contrast caused by insufficient light sources. Additionally, it can also handle detail noise such as ink bleeding. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a general flowchart of a method for enhancing photographed document images according to an embodiment of the present invention;
[0046] Figure 2 This is a partial sample diagram of manually labeled training data in an embodiment of the present invention;
[0047] Figure 3 This is a schematic diagram of some enhancement results in the embodiments of the present invention, including a first document image, a shadow image, a second document image, and a third document image;
[0048] Figure 4 This is a flowchart illustrating the steps of a photographic document image enhancement method according to an embodiment of the present invention. Detailed Implementation
[0049] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0050] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0051] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.
[0052] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0053] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0054] like Figure 4 As shown, this embodiment provides a method for enhancing photographed document images, including the following steps:
[0055] S1. Obtain the first document image and the shadow image corresponding to the first document image;
[0056] S2. Perform illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image;
[0057] S3. After concatenating the first document image and the second document image along the channel dimension, input them into a preset second deep convolutional neural network for enhancement processing, and output a third document image as the final enhancement result.
[0058] The above method will be explained in detail below with reference to specific embodiments and accompanying drawings.
[0059] like Figure 1 As shown, this embodiment provides a method for enhancing photographic document images, used to enhance photographic document images containing complex degradation in real-world scenes. The method specifically includes the following steps:
[0060] Step 1: Acquire the first document image, which is a document image captured by a camera (such as a smart terminal, camera, etc.). Scale the first document image to a uniform size of 512*512 and input it into the first deep convolutional neural network. The first convolutional neural network predicts and outputs the shadow map of the first document image and scales it back to the original size. Scaling it to a smaller 512*512 size allows the model to have a better global field of view, thus better recognizing the global shadow distribution.
[0061] Step 2: Divide the first document image and the shadow map to obtain the second document image after illumination correction; this step is based on the Lambertian reflection model, that is, the image can be decomposed into a shadow map and a reflection map: Where R is the reflection map, which physically represents the reflectivity of the object's surface; S is the shadow map, which physically represents the shadow cast by light. This involves pixel-by-pixel multiplication. Based on this reflection model, we treat R as the second document image after illumination correction, and I as the first document image. Therefore, based on the shadow map and the first document image, we can obtain the second document image R = I / S.
[0062] Step 3: The second document image can effectively correct global illumination, but it cannot enhance high-frequency details. To further enhance local details, a second deep convolutional neural network is used. The second deep convolutional neural network takes the concatenated result of the first and second document images as input and outputs the final enhanced result.
[0063] In some optional embodiments, step 1 employs a first deep convolutional neural network to predict the shadow map, and the network parameters are trained and optimized using a manually collected and labeled dataset. Specifically, the network parameter optimization includes:
[0064] (1) Data Acquisition: 450 photographed document images were collected from public data and the Internet. These images were then enhanced using the professional image processing software Photoshop. Based on the Lambertian reflection model, the photographed document image and the enhanced document image were divided to obtain the corresponding shadow map, which served as the label for model training.
[0065] (2) Network training:
[0066] (2-1) Constructing a deep neural network: Use the UNeXt segmentation model as the network structure, and set the number of output channels to 3, that is, the number of channels in the output result of the last layer of the network is 3.
[0067] (2-2) Training method: Gradient descent algorithm is used for training. The gradient is calculated from the last layer and passed layer by layer to update all parameters, thus achieving the purpose of training the network. The loss function during training is L1 loss.
[0068] (2-3) Setting training parameters:
[0069] Number of iterations: 100 epochs
[0070] Optimizer: Adam
[0071] Learning rate: 0.0001 (Learning rate update strategy: after every 30 iterations, the learning rate decays to 1 / 10 of its original value)
[0072] Weight decay: 0.0005
[0073] (2-4) Start training the deep neural network with randomly initialized parameters.
[0074] (2-5) Optimization objective: Minimize the L1 distance between the model output and the shadow map label.
[0075] In some optional embodiments, step 3 employs a second deep convolutional neural network to obtain the final enhancement. The network parameters are pre-trained and optimized using manually labeled data, specifically including:
[0076] (1) Data Acquisition: 450 photographic document images were collected from publicly available data and the Internet. See [link to relevant documentation]. Figure 2 The document image was enhanced using professional image processing software Photoshop, and the enhanced image was used as the label for model training. The model input consisted of the photographed document image and the output of the trained first deep convolutional neural network.
[0077] (2) Network training:
[0078] (2-1) Constructing a deep neural network: The UNeXt segmentation model is used as the network structure, with the number of output channels set to 3, meaning the last layer of the network outputs has 3 channels. This is to obtain multi-scale outputs. like Figure 1 As shown, after each skip connection in UNeXt, an output layer consisting of a 1x1 convolution and a sigmoid pair is added. After the model training is complete, The output layer can be removed, leaving only the output layer. The output layer.
[0079] (2-3) Setting training parameters:
[0080] Number of iterations: 100 epochs
[0081] Optimizer: Adam
[0082] Learning rate: 0.0001 (Learning rate update strategy: after every 30 iterations, the learning rate decays to 1 / 10 of its original value)
[0083] Weight decay: 0.0005
[0084] (2-4) Start training the deep neural network with randomly initialized parameters.
[0085] (2-5) Optimization objective: Minimize Maximize the L1 distance to the corresponding label. Minimize the SSIM metric for the corresponding label. The total variation regularization term.
[0086] like Figure 3 As shown, the method provided in this embodiment can process photographed document images with complex degradation, and can achieve good enhancement effects. In summary, the method proposed in this embodiment can handle various lighting degradations, including shadows caused by objects blocking the light source, shadows caused by uneven light sources, shadows caused by uneven paper, and low contrast caused by insufficient light sources; in addition, it can also handle detailed noise such as text bleeding.
[0087] This embodiment also provides an image enhancement system for photographed documents, including:
[0088] An image acquisition module is used to acquire a first document image and to acquire a shadow image corresponding to the first document image;
[0089] The image correction module is used to perform illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image;
[0090] The enhancement processing module is used to concatenate the first document image and the second document image along the channel dimension, input them into a preset second deep convolutional neural network for enhancement processing, and output a third document image as the final enhancement result.
[0091] This embodiment of the photographic document image enhancement system can execute the photographic document image enhancement method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0092] This embodiment also provides an image enhancement device for photographed documents, including:
[0093] At least one processor;
[0094] At least one memory for storing at least one program;
[0095] When the at least one program is executed by the at least one processor, the at least one processor performs the following: Figure 4 The method shown.
[0096] This embodiment of the photographic document image enhancement device can execute the photographic document image enhancement method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0097] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 4 The method shown.
[0098] This embodiment also provides a storage medium storing instructions or programs that can execute the image enhancement method for photographed documents provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.
[0099] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.
[0100] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0101] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0102] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0103] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0104] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0105] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0106] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0107] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A method for enhancing photographed document images, characterized in that, Includes the following steps: Obtain the first document image and the shadow image corresponding to the first document image; The second document image is obtained by performing illumination correction processing based on the first document image and the obtained shadow map; The first document image and the second document image are concatenated along the channel dimension and then input into a preset second deep convolutional neural network for enhancement processing, outputting a third document image as the final enhancement result. The step of obtaining the shadow image corresponding to the first document image includes: The first document image is scaled to a preset resolution and input into a preset first deep convolutional neural network to output a shadow map corresponding to the first document image. The resolution of the shadow image is scaled to make the resolution of the shadow image consistent with the resolution of the first document image; The first deep convolutional neural network was obtained by training in the following way: Acquire multiple photographic document images, perform enhancement processing on the photographic document images, and obtain an enhanced dataset; Based on the Lambertian reflection model, the images in the augmented dataset are processed to obtain the corresponding shadow maps, which serve as labels for model training; a training set is constructed based on the augmented dataset and the obtained shadow maps. The first deep convolutional neural network was constructed using the UNeXt segmentation model as the network structure. The first deep convolutional neural network is trained using the training set to obtain the trained first deep convolutional neural network; The loss function used during training is L1 loss; The training phase of the second deep convolutional neural network employs a strategy of multi-scale output and multi-loss function supervision. Among them, the multi-scale output strategy uses multiple convolutional output layers, takes the intermediate layer features as input, and predicts multiple outputs with different resolutions. The additional convolutional output layers are only needed during the training phase and are discarded during the forward testing phase. The multi-loss function strategy designs different loss functions for supervision based on the output at different resolutions.
2. The method for enhancing photographic document images according to claim 1, characterized in that, The step of performing illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image includes: Perform a pixel-level division operation between the first document image and the obtained shadow image to obtain the second document image after illumination correction.
3. The method for enhancing photographic document images according to claim 2, characterized in that, The second document image is obtained by processing it using the following formula: in, The first document image, This is a shaded image.
4. The method for enhancing photographic document images according to claim 1, characterized in that, The second deep convolutional neural network was obtained by training in the following way: Multiple photographed document images are acquired, and the photographed document images are enhanced using preset image processing software to obtain enhanced document images as labels for model training; a training set is constructed based on the photographed document images and labels; The second deep convolutional neural network is constructed using the UNeXt segmentation model as the network structure; wherein the input of the second deep convolutional neural network consists of the photographed document image and the shadow map corresponding to the photographed document image; The constructed second deep convolutional neural network is trained using the training set to obtain the trained second deep convolutional neural network; Among them, in order to obtain multi-scale output In UNeXt, an output layer is connected after each skip connection. After the model training is complete, The output layer can be removed, leaving only the output layer. The output layer.
5. A photographic document image enhancement system, used to implement the method according to any one of claims 1-4, characterized in that, include: An image acquisition module is used to acquire a first document image and to acquire a shadow image corresponding to the first document image; The image correction module is used to perform illumination correction processing based on the first document image and the obtained shadow map to obtain the second document image; The enhancement processing module is used to concatenate the first document image and the second document image along the channel dimension, input them into a preset second deep convolutional neural network for enhancement processing, and output a third document image as the final enhancement result.
6. An image enhancement device for photographed documents, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-4.
7. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-4.