A unified method and system for document image enhancement

Abstract

The present invention discloses a unified method and system for document image enhancement, wherein the method comprises the following steps: extracting two-dimensional visual cues based on the document image to be enhanced and the specific enhancement task to be performed; using the extracted two-dimensional visual cues to guide a deep neural network with shared parameters to perform the specific enhancement task on the document image to be enhanced. The present invention innovatively proposes to use underlying visual features as visual cues to guide the model to perform different enhancement tasks, thereby improving the enhancement performance while achieving the goal of using only one model and training once to complete multiple document image enhancement tasks.

Description

Technical Field

[0001] The invention belongs to the technical field of pattern recognition and artificial intelligence, and specifically relates to a unified method and system for document image enhancement. Background Art

[0002] Existing document image enhancement methods usually only consider a single enhancement task, which makes the document image enhancement system too complicated. In practical applications, it is usually necessary to maintain multiple different models, which increases manpower and material costs. Although some methods in recent years have proposed to use the same model structure to complete multiple document image enhancement tasks, simplifying the model design, they still need to perform separate training for different tasks, that is, multiple tasks and multiple models, which is still not conducive to practical applications. The underlying visual features are often used to assist in improving the performance of document image enhancement models. Therefore, there is an urgent need for a unified method and system for document image enhancement. Summary of the invention

[0003] In view of the shortcomings of the existing technology, the present invention proposes a unified method and system for document image enhancement, and innovatively proposes to use underlying visual features as visual cues to guide the model to perform different enhancement tasks. While improving the enhancement performance, it is possible to complete multiple document image enhancement tasks with only one model trained once.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A unified method for document image enhancement comprises the following steps:

[0006] Extracting two-dimensional visual cues based on the document image to be enhanced and the specific enhancement task to be performed;

[0007] The extracted 2D visual cues are used to guide a deep neural network with shared parameters to perform specific enhancement tasks on the document images to be enhanced.

[0008] Preferably, the specific enhancement tasks are geometric correction, shadow removal, appearance enhancement, deblurring and binarization, respectively.

[0009] Preferably, the extracted two-dimensional visual cues include: two-dimensional visual cues for correction tasks, two-dimensional visual cues for shadow removal tasks, two-dimensional visual cues for appearance enhancement tasks, two-dimensional visual cues for deblurring tasks, and two-dimensional visual cues for binarization tasks;

[0010] The calculation method of the two-dimensional visual prompt of the correction task is:

[0011] Correction task 2D visual cue = [P m (I s ), P cx , P cy], where the [] symbol indicates concatenation in the channel dimension, P m () represents the segmentation model, P cx , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s represents the document image to be enhanced;

[0012] The calculation method of the two-dimensional visual cue of the shadow removal task is:

[0013] Shadow removal task 2D visual cue = P bg (Is), where P bg () represents the process of extracting low-frequency background of the image;

[0014] The calculation method of the two-dimensional visual cue of the appearance enhancement task is:

[0015] 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (Is)), where abs() is an absolute value operation;

[0016] The calculation method of the two-dimensional visual cue of the deblurring task is:

[0017] Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (Is)], where P g () represents the process of extracting image gradient using Sobel operator;

[0018] The calculation method of the two-dimensional visual cue of the binary task is:

[0019] Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

[0020] Preferably, the method of using the extracted two-dimensional visual cues to guide the deep neural network to perform a specific enhancement task on the document image to be enhanced includes:

[0021] The extracted two-dimensional visual cues and the document image to be enhanced are concatenated in the channel dimension to obtain an input data of size H×W×6;

[0022] Inputting the input data into a deep neural network for underlying image enhancement for enhancement;

[0023] Among them, the deep neural network uses the Restormer underlying image enhancement model.

[0024] The present invention also provides a unified document image enhancement system, comprising: an extraction module and an enhancement module;

[0025] The extraction module is used to extract two-dimensional visual cues based on the document image to be enhanced and the specific enhancement task to be performed;

[0026] The enhancement module is used to utilize the extracted two-dimensional visual cues to guide the deep neural network with shared parameters to perform a specific enhancement task on the document image to be enhanced.

[0027] Preferably, the specific enhancement tasks are geometric correction, shadow removal, appearance enhancement, deblurring and binarization, respectively.

[0028] Preferably, the extracted two-dimensional visual cues include: two-dimensional visual cues for correction tasks, two-dimensional visual cues for shadow removal tasks, two-dimensional visual cues for appearance enhancement tasks, two-dimensional visual cues for deblurring tasks, and two-dimensional visual cues for binarization tasks;

[0029] The calculation process of the two-dimensional visual cue of the correction task is as follows:

[0030] Correction task 2D visual cue = [P m (I s ), P cx , P cy ], where the [] symbol indicates concatenation in the channel dimension, P m () represents the segmentation model, P cx , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s represents the document image to be enhanced;

[0031] The calculation process of the two-dimensional visual cue of the shadow removal task is:

[0032] Shadow removal task 2D visual cue = P bg (I s ), where P bg () represents the process of extracting low-frequency background of the image;

[0033] The calculation process of the two-dimensional visual cue of the appearance enhancement task is:

[0034] 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (I s )), where abs() is the absolute value operation;

[0035] The calculation process of the two-dimensional visual cue of the deblurring task is:

[0036] Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (I s )], where P g () represents the process of extracting image gradient using Sobel operator;

[0037] The calculation process of the two-dimensional visual cue of the binary task is:

[0038] Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

[0039] Preferably, the enhancement module comprises: a splicing unit and an enhancement unit;

[0040] The splicing unit is used to splice the extracted two-dimensional visual cue and the document image to be enhanced in the channel dimension to obtain an input data with a size of H×W×6;

[0041] The enhancement unit is used to input the input data into a deep neural network for underlying image enhancement for enhancement;

[0042] Among them, the deep neural network uses the Restormer underlying image enhancement model.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention extracts multiple visual features of the image to be enhanced to form visual cues for different tasks, which are used to prompt the deep neural network to perform different enhancement tasks while also providing additional information of the image to be enhanced to improve the enhancement performance. It can be achieved that a variety of document image enhancement tasks can be completed with one model and one training, effectively reducing the complexity of the document image enhancement system and reducing maintenance costs. In addition, the two-dimensional visual cues proposed by the present invention are some underlying visual prior features that are easy to extract and have the same resolution as the image to be enhanced. This makes it possible to flexibly select a variety of commonly used image enhancement deep neural networks when constructing a multi-task unified enhancement model. For example, the two-dimensional visual cues and the image to be enhanced can be input into a commonly used image enhancement deep neural network by simply splicing them in the channel dimension without the need for additional structural modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 is an overall flow chart of a unified method for document image enhancement according to an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of different two-dimensional visual prompts designed for different tasks in an embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the test effect when the unified model obtained in the embodiment of the present invention performs different enhancement tasks on the same input image. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Embodiment 1

[0052] like Figure 1 As shown, the present invention provides a unified method for document image enhancement, comprising the following steps:

[0053] S1: Extracting two-dimensional visual cues based on the document image to be enhanced and the specific enhancement task to be performed;

[0054] S2: Leverage the extracted 2D visual cues to guide a deep neural network with shared parameters to perform a specific enhancement task on the enhanced document image.

[0055] In this embodiment, the two-dimensional visual cues described in S1 are some underlying visual features extracted from the document image to be enhanced, including but not limited to: gradient map, low-frequency background map, document area foreground segmentation map, preliminary binarization result map, binarization threshold map, etc.

[0056] The two-dimensional visual cues for different tasks described in S1 are composed of different underlying visual prior features, and the obtained two-dimensional visual cues have sufficient discriminability, and the deep neural network learning with shared parameters of S2 can distinguish the tasks to be performed based on this.

[0057] The two-dimensional visual prompt described in S1 selects appropriate visual feature composition according to the characteristics of different tasks, and can be used as additional information of the image to be enhanced to assist the deep neural network in S2 to perform enhancement, so as to achieve the purpose of improving the enhancement effect.

[0058] The visual features described in S1 are two-dimensional, consistent with the resolution of the image to be enhanced, and can be efficiently integrated into a variety of mature deep neural networks for underlying image enhancement without the need for additional network structure design. The input method of two-dimensional visual cues includes but is not limited to directly splicing the image to be enhanced in the channel dimension.

[0059] S2 can achieve a variety of document image enhancement tasks using only one deep neural network with shared parameters and training once. The enhancement tasks include but are not limited to: geometric correction, shadow removal, appearance enhancement, binarization, deblurring, etc.

[0060] Specifically, this embodiment provides a unified method for document image enhancement, which realizes one model to unify five document image enhancement tasks, namely geometric correction, shadow removal, appearance enhancement, deblurring and binarization. The method specifically includes the following steps:

[0061] S1: Figure 1 and Figure 2 As shown, for a given document image to be enhanced I s(size is H×W×3) and the enhancement task to be performed, this embodiment extracts different underlying visual features according to the characteristics of the task and combines them to form the visual prompts of the specified task, and the visual prompts of all tasks are two-dimensional images with the same resolution as the image to be corrected and the number of channels is 3. The extracted two-dimensional visual prompts include: two-dimensional visual prompts for correction tasks, two-dimensional visual prompts for shadow removal tasks, two-dimensional visual prompts for appearance enhancement tasks, two-dimensional visual prompts for deblurring tasks, and two-dimensional visual prompts for binarization tasks.

[0062] For the correction task, the document segmentation mask is a commonly used auxiliary feature that can be used to improve the performance of the correction model. We use the document segmentation mask as one of the channels of visual cues. The segmentation mask is implemented using the trained segmentation model provided in the existing document image correction method Marrior, denoted as P m (). In addition, considering that document correction is a coordinate-related regression task, the model needs to learn to understand the coordinates, so we take the horizontal and vertical coordinate values ​​P of the pixel location as cx (i, j) = i, P cy (i, j) = j as the remaining two channels of visual cues (the horizontal and vertical coordinates are both H×W), and finally we can get:

[0063] Correction task 2D visual cue = [P m (I s ), P cx , P cy ], where [] means splicing in the channel dimension (the same below), P m () represents the segmentation model, P ex , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s Represents the document image to be enhanced.

[0064] For the shadow removal task, this embodiment uses low-frequency background as a visual cue. This visual feature is also used in many shadow removal methods. This embodiment first scales the image to be enhanced to a size of 1024×1024×3 and then performs a morphological opening operation (kernel size is 7x7) to remove the high-frequency foreground. The result is then subjected to a median filter with a kernel size of 21 to remove noise. The complete process is recorded as P bg (). Finally, we can get:

[0065] Shadow removal task 2D visual cue = P bg (I s ), where P bg () represents the process of extracting low-frequency background of the image.

[0066] The appearance enhancement task uses the high-frequency foreground after removing the low-frequency background as a visual cue.bg () to build:

[0067] 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (I s )), where abs() is the absolute value operation.

[0068] For the deblurring task, the three channels of visual cues are all gradient maps of the input image. The gradient map is obtained by edge extraction using the Sobel operator, denoted as P g (). Finally, we can get:

[0069] Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (I s )], where P g () represents the process of extracting image gradient using Sobel operator.

[0070] For the binarization task, many methods have proved that cascade binarization can achieve better results, that is, a preliminary binarization is performed first to provide guidance information for the second binarization. This embodiment uses the Sauvola adaptive binarization method to extract the preliminary binarization results P b (I s ) and the local threshold map P t (I s ) are used as two channels of visual cues, and the image gradient P g (I s ) as the third channel. Finally, we can get:

[0071] Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

[0072] In this embodiment, S2: the method of using the extracted two-dimensional visual cues to guide the deep neural network to perform a specific enhancement task on the document image to be enhanced includes:

[0073] The two-dimensional visual cue obtained in S1 is concatenated with the image to be enhanced in the channel dimension to obtain an input of size H×W×6, which is then input into the deep neural network for underlying image enhancement for enhancement. The deep neural network used in step S2 is Restormer, and the parameters of the network are trained based on the public datasets of each task. The network parameter optimization specifically includes:

[0074] (1) Data acquisition: Correction task dataset: Doc3D; Shadow removal dataset: FSDSRD, RDD; Appearance enhancement dataset: RealDAE; Deblurring dataset: TDD; Binarization dataset: (H)-DIBCO past competition datasets, Synchromedia Multispectral, Persian Heritage Image Binarization, Bickley Diary.

[0075] (2) Data processing: Based on the above visual cue design scheme, the visual cues of each task data are extracted and concatenated with the image to be enhanced in the channel dimension to obtain a new input.

[0076] (3) Network training: The training process of the deep neural network with shared parameters includes: extracting corresponding two-dimensional visual cues from open source data of different tasks and training on multi-task mixed data.

[0077] (3-1) Construct a deep neural network: Select the Restormer underlying image enhancement model as the network structure, and set the number of input layer channels to 6;

[0078] (3-2) Training method: The training uses the gradient descent algorithm, which calculates the gradient from the last layer, passes it layer by layer, and updates all parameters to achieve the purpose of training the network. During training, the cross entropy loss function is used for the binarization task, and the L2 loss function is used for the other tasks;

[0079] (3-3) Training parameter settings: Iterations: 100 epochs; Optimizer: Adam; Learning rate: 0.0001 (learning rate update strategy: after every 30 iterations, the learning rate decays to 1 / 10 of the original); Weight decay: 0.0005;

[0080] (3-4) Start training the deep neural network with random initialization parameters;

[0081] (3-5) Optimization objective: minimize the L2 loss function.

[0082] like Figure 3The figure shows a schematic diagram of the test effect of the unified model when performing different enhancement tasks on the same input image.

[0083] Embodiment 2

[0084] The present invention also provides a unified document image enhancement system, comprising: an extraction module and an enhancement module;

[0085] The extraction module is used for extracting two-dimensional visual cues based on the document image to be enhanced and the specific enhancement task to be performed;

[0086] The enhancement module is used to utilize the extracted two-dimensional visual cues to guide the deep neural network to perform specific enhancement tasks on the enhanced document image.

[0087] In this embodiment, the specific enhancement tasks are geometric correction, shadow removal, appearance enhancement, deblurring, and binarization.

[0088] In this embodiment, the extracted two-dimensional visual cues include: two-dimensional visual cues for the correction task, two-dimensional visual cues for the shadow removal task, two-dimensional visual cues for the appearance enhancement task, two-dimensional visual cues for the deblurring task, and two-dimensional visual cues for the binarization task;

[0089] The calculation process of the two-dimensional visual cue of the correction task is as follows:

[0090] Correction task 2D visual cue = [P m (I s ), P cx , P cy ], where the [] symbol indicates concatenation in the channel dimension, P m () represents the segmentation model, P cx , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s represents the document image to be enhanced;

[0091] The calculation process of the two-dimensional visual cue of the shadow removal task is:

[0092] Shadow removal task 2D visual cue = P bg (I s ), where P bg () represents the process of extracting low-frequency background of the image;

[0093] The calculation process of the two-dimensional visual cue of the appearance enhancement task is:

[0094] 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (Is)), where abs() is an absolute value operation;

[0095] The calculation process of the two-dimensional visual cue of the deblurring task is:

[0096] Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (I s )], where P g () represents the process of extracting image gradient using Sobel operator;

[0097] The calculation process of the two-dimensional visual cue of the binary task is:

[0098] Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

[0099] Preferably, the enhancement module comprises: a splicing unit and an enhancement unit;

[0100] The splicing unit is used to splice the extracted two-dimensional visual cue and the document image to be enhanced in the channel dimension to obtain an input data with a size of H×W×6;

[0101] The enhancement unit is used to input the input data into a deep neural network for underlying image enhancement for enhancement;

[0102] Among them, the deep neural network uses the Restormer underlying image enhancement model.

[0103] Embodiment 3

[0104] This embodiment provides a unified device for document image enhancement, including:

[0105] at least one processor;

[0106] at least one memory for storing at least one program;

[0107] When the at least one program is executed by the at least one processor, the at least one processor implements the method described in embodiment one.

[0108] Embodiment 4

[0109] This embodiment provides a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to perform the method described in the first embodiment.

[0110] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0111] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A unified method for document image enhancement, characterized in that: The following steps are involved: Extracting two-dimensional visual cues based on the document image to be enhanced and the enhancement task to be performed; The extracted 2D visual cues are used to guide a deep neural network with shared parameters to perform enhancement tasks on the document image to be enhanced. The extracted two-dimensional visual cues include: two-dimensional visual cues for the correction task, two-dimensional visual cues for the shadow removal task, two-dimensional visual cues for the appearance enhancement task, two-dimensional visual cues for the deblurring task, and two-dimensional visual cues for the binarization task; The calculation method of the two-dimensional visual prompt of the correction task is: Correction task 2D visual cue = [P m (I s ), P cx , P cy ], where the [] symbol indicates concatenation in the channel dimension, P m () represents the segmentation model, P cx , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s represents the document image to be enhanced; The calculation method of the two-dimensional visual cue of the shadow removal task is: Shadow removal task 2D visual cue = P bg (I s ), where P bg () represents the process of extracting low-frequency background of the image; The calculation method of the two-dimensional visual cue of the appearance enhancement task is: 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (I s )), where abs() is the absolute value operation; The calculation method of the two-dimensional visual cue of the deblurring task is: Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (I s )], where P g () represents the process of extracting image gradient using Sobel operator; The calculation method of the two-dimensional visual cue of the binary task is: Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

2. The unified method for document image enhancement according to claim 1, characterized in that: The enhancement tasks are geometric correction, shadow removal, appearance enhancement, deblurring, and binarization.

3. The unified method for document image enhancement according to claim 1, characterized in that: Methods for using the extracted two-dimensional visual cues to guide a deep neural network to perform an enhancement task on a document image to be enhanced include: The extracted two-dimensional visual cues and the document image to be enhanced are concatenated in the channel dimension to obtain an input data of size H×W×6; Inputting the input data into a deep neural network for underlying image enhancement for enhancement; Among them, the deep neural network uses the Restormer underlying image enhancement model.

4. A unified document image enhancement system, characterized in that: include: Extraction module and enhancement module; The extraction module is used to extract two-dimensional visual cues based on the document image to be enhanced and the enhancement task to be performed; The enhancement module is used to use the extracted two-dimensional visual cues to guide the deep neural network with shared parameters to perform the enhancement task on the document image to be enhanced; The extracted two-dimensional visual cues include: two-dimensional visual cues for the correction task, two-dimensional visual cues for the shadow removal task, two-dimensional visual cues for the appearance enhancement task, two-dimensional visual cues for the deblurring task, and two-dimensional visual cues for the binarization task; The calculation process of the two-dimensional visual cue of the correction task is as follows: Correction task 2D visual cue = [P m (I s ), P cx , P cy ], where the [] symbol indicates concatenation in the channel dimension, P m () represents the segmentation model, P cx , P cy Indicates the horizontal and vertical coordinate values ​​of the pixel location, I s represents the document image to be enhanced; The calculation process of the two-dimensional visual cue of the shadow removal task is: Shadow removal task 2D visual cue = P bg (I s ), where P bg () represents the process of extracting low-frequency background of the image; The calculation process of the two-dimensional visual cue of the appearance enhancement task is: 2D visual cue for appearance enhancement task = 255-abs(I s -P bg (I s )), where abs() is the absolute value operation; The calculation process of the two-dimensional visual cue of the deblurring task is: Deblurring task 2D visual cue = [P g (I s ), P g (I s ), P g (I s )], where P g () represents the process of image gradient extraction using Sobel operator; The calculation process of the two-dimensional visual cue of the binary task is: Binarization task Two-dimensional visual cue = [P b (I s ), P t (I s ), P g (I s )], where P b (I s ) represents the initial binarization result, P t (I s ) represents the local threshold map, P g (I s ) represents the image gradient.

5. The document image enhancement unified system according to claim 4, characterized in that: The enhancement tasks are geometric correction, shadow removal, appearance enhancement, deblurring, and binarization.

6. The document image enhancement unified system according to claim 4, characterized in that: The enhancement module includes: a splicing unit and an enhancement unit; The splicing unit is used to splice the extracted two-dimensional visual cue and the document image to be enhanced in the channel dimension to obtain an input data with a size of H×W×6; The enhancement unit is used to input the input data into a deep neural network for underlying image enhancement for enhancement; Among them, the deep neural network uses the Restormer underlying image enhancement model.

Images