Document image processing method and device, training sample generation method and device

Abstract

The present disclosure provides a document image processing method and apparatus, and a training sample generation method and apparatus. The method comprises: determining an initial character region containing characters in a document image to be processed; optimizing the initial character region to determine the character boundaries, and determining an optimized target character region based on the character boundaries; removing the target character region from the document image to be processed, and generating a lighting image based on the document image to be processed without the target character region, wherein the lighting image is used to reflect ambient lighting information. This generates a lighting image containing real lighting information, which serves as the basis for training sample generation, resolving the current difficulty in obtaining lighting information.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for processing document images, and a method and device for generating training samples. Background Art

[0002] Deep learning algorithms are widely used in computer vision and have achieved impressive results in many areas. Using deep learning algorithms to enhance the lighting of document images and eliminate shadows introduced by photography is a novel and feasible solution. However, models built based on deep learning rely on sample data and are highly data-driven. The quantity, quality, and diversity of the sample data used to train the model directly impact its accuracy.

[0003] Currently, document image data often relies heavily on manual collection, requiring data collectors to capture a large number of different document images for model training. Furthermore, to obtain corresponding labels, document images of the same scene must be captured under dark conditions. Image registration and information labeling must then be performed on the illuminated and dark document images. This workload is substantial, and sample data acquisition has become a significant bottleneck for the application of deep learning to document image enhancement tasks. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a method and device for processing document images, and a method and device for generating training samples.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing a document image is provided, the method comprising:

[0006] Determine an initial character region where characters are located in the document image to be processed;

[0007] Optimizing the initial character region, determining the boundaries of the characters, and determining an optimized target character region based on the boundaries of the characters;

[0008] The target character area is removed from the document image to be processed, and a lighting image is generated based on the document image to be processed from which the target character area is removed. The lighting image is used to reflect lighting information of the environment.

[0009] Optionally, determining the initial character area in the document image to be processed includes:

[0010] performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image;

[0011] Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

[0012] Optionally, optimizing the initial character area includes:

[0013] The initial character region is optimized based on a morphological algorithm.

[0014] Optionally, generating a lighting image based on the document image to be processed from which the target character area is removed includes:

[0015] For each pixel point in the blank area, the pixel values ​​of adjacent pixels within a preset range from the pixel point are fitted based on an interpolation algorithm, and the pixel values ​​are filled at the pixel point. The blank area is the area of ​​the document image to be processed from which the target character area is removed.

[0016] The document image to be processed that has been filled with pixel values ​​is determined as the illumination image.

[0017] Optionally, the difference between the pixel value of the background area in the document image to be processed and the pixel value of the character area in the document image to be processed is greater than a pixel threshold.

[0018] According to a second aspect of an embodiment of the present disclosure, a method for generating a training sample is provided, the method comprising:

[0019] Obtain document image samples;

[0020] superimposing a light image on the document image sample to obtain a superimposed image, wherein the light image is obtained by any of the above-mentioned document image processing methods;

[0021] The superimposed image and the illumination image are used as a training sample pair for model training, and the model is used to eliminate illumination information in an input document image.

[0022] According to a third aspect of an embodiment of the present disclosure, there is provided a document image processing device, the processing device comprising:

[0023] A determination module, used for determining an initial character region where characters are located in a document image to be processed;

[0024] an optimization module, configured to optimize the initial character region, determine the boundaries of the characters, and determine an optimized target character region based on the boundaries of the characters;

[0025] The generating module is used to remove the target character area from the document image to be processed, and generate a lighting image based on the document image to be processed with the target character area removed, wherein the lighting image is used to reflect the lighting information of the environment.

[0026] Optionally, when optimizing the initial character area, the optimization module is configured to:

[0027] performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image;

[0028] Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

[0029] Optionally, when generating the illumination image based on the document image to be processed from which the target character area is removed, the generating module is configured to:

[0030] For each pixel point in the blank area, the pixel values ​​of adjacent pixels within a preset range from the pixel point are fitted based on an interpolation algorithm, and the pixel values ​​are filled at the pixel point. The blank area is the area of ​​the document image to be processed from which the target character area is removed.

[0031] The document image to be processed that has been filled with pixel values ​​is determined as the illumination image.

[0032] According to a fourth aspect of an embodiment of the present disclosure, a device for generating a training sample is provided, the device comprising:

[0033] An acquisition module, used for acquiring document image samples;

[0034] a superimposition module, configured to superimpose a light image on the document image sample to obtain a superimposed image, wherein the light image is obtained by the document image processing device described in any one of the above items;

[0035] The sample generation module is used to use the superimposed image and the illumination image as a training sample pair for model training, and the model is used to eliminate illumination information in the input document image.

[0036] Optionally, the document image sample includes at least one of:

[0037] a first document image converted from an electronic document;

[0038] A second document image is obtained by performing spatial geometric transformation on the first document image.

[0039] According to a fifth aspect of an embodiment of the present disclosure, there is provided a document image processing device, comprising:

[0040] processor;

[0041] a memory for storing processor-executable instructions;

[0042] Wherein, the processor is configured to implement the steps of any one of the above-mentioned document image processing methods when executing the execution instruction.

[0043] According to a sixth aspect of an embodiment of the present disclosure, there is provided a device for generating a training sample, including:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to implement any one of the steps of the above-mentioned method for generating training samples when executing the execution instruction.

[0047] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0048] In the embodiment of the present disclosure, by performing binarization and morphological processing on the document image, the boundaries of the character areas in the document image can be accurately located, and an illumination image including illumination information is generated based on the document image after removing the character areas along the boundaries, which serves as the basis for generating training samples, thereby solving the current problem of difficulty in obtaining illumination information.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] Figure 1 The figure is a flowchart of a method for generating training samples according to an exemplary embodiment of the present disclosure.

[0052] Figure 2 A document image processing method according to an exemplary embodiment of the present disclosure is shown.

[0053] Figure 3a 3 is a schematic diagram of a document image obtained by photographing a paper document according to an exemplary embodiment of the present disclosure.

[0054] Figure 3b FIG. 1 is a schematic diagram showing an initial character area according to an exemplary embodiment of the present disclosure.

[0055] Figure 3c This disclosure is based on an exemplary embodiment of the present invention. Figure 3b Schematic diagram of the results after the corrosion operation.

[0056] Figure 3d This disclosure is based on an exemplary embodiment of the present invention. Figure 3c Schematic diagram of the result after the expansion operation.

[0057] Figure 3e 3 is a schematic diagram of a document image after removing character areas according to an exemplary embodiment of the present disclosure.

[0058] Figure 4 Another document image processing method according to an exemplary embodiment of the present disclosure is shown.

[0059] Figure 5 The figure is a module diagram of a document image processing device according to an exemplary embodiment of the present disclosure.

[0060] Figure 6 It is a module diagram of another document image processing device according to an exemplary embodiment of the present disclosure.

[0061] Figure 7 It is a module diagram of a device for generating training samples according to an exemplary embodiment of the present disclosure.

[0062] Figure 8 The present invention is a block diagram of a device for document image processing according to an exemplary embodiment of the present invention.

[0063] Figure 9 is a block diagram showing another device for document image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0065] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0066] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

[0067] With the rapid development of portable photography devices such as smartphones, people can now capture large quantities of high-quality photos more conveniently. In addition to capturing the wonderful moments of daily life, people also want to be able to capture documents anytime, anywhere and obtain standardized document images for recording and sharing important information. Compared to traditional scanners, generating digital documents with mobile phone photography is easy to use and simple to operate. However, due to the complex and changing shooting environment and the influence of lighting conditions, the quality of captured document images is often not high, making efficient and robust document image enhancement a pressing problem.

[0068] Deep learning algorithms are widely used in the field of computer vision and have achieved remarkable results in many areas. Using deep learning algorithms to enhance document images to improve lighting and eliminate shadows introduced by photography is a novel and feasible solution. However, because deep learning algorithms rely on sample data and are highly data-driven, the quantity, quality, and diversity of the sample data used to train the model directly impact its accuracy.

[0069] Currently, training sample data for image enhancement models often relies heavily on manual collection. Collectors need to capture a large number of document images with varying lighting conditions, content, and angles to meet the requirements for model training. At the same time, to obtain corresponding training labels, document images of the same scene must be captured without lighting. Because the distances and angles of the two shots differ, both document images captured under lighting conditions (those with lighting information) and those captured without lighting conditions (those without lighting information) require image registration and alignment, as well as lighting information annotation. Only then can the document images with and without lighting information, which have undergone image registration and information annotation, be used as image sample pairs for training the model.

[0070] The acquisition process of the above sample data has very strict requirements on the shooting environment. When actually shooting images with and without lighting information, it is difficult to ensure that the shooting environment conditions of the two shots are completely consistent. On the other hand, the captured images need to be registered and the lighting information labeled. The calculation amount is large, and the production of sample data is very cumbersome, making it impossible to obtain batch data in a short time.

[0071] Based on the above problems, an embodiment of the present disclosure provides a method for generating training samples, which superimposes a real lighting image extracted from an image on a document image without lighting information as a training sample for model training, so as to reduce the workload of manual collection and information labeling.

[0072] Next, the embodiments of the present disclosure are described in detail.

[0073] like Figure 1 As shown, Figure 1 FIG. 1 is a flowchart of a method for generating a training sample according to an exemplary embodiment of the present disclosure, comprising the following steps:

[0074] Step 101: Obtain a document image sample.

[0075] Model training requires a large number of training samples. Document images, as the basis for generating training samples, can be obtained by, but are not limited to, converting a large number of electronic documents. For example, converting electronic documents in formats such as Word, PDF, and TXT to image formats to obtain document image samples, which do not contain lighting information. To meet the model training requirement for diverse training samples, electronic documents can include various electronic magazines, electronic journals, and electronic books.

[0076] Step 102: superimpose the illumination image on the document image sample to obtain a superimposed image.

[0077] In step 102 , for each document image sample obtained in step 101 , an illumination image needs to be superimposed.

[0078] The illumination image can reflect the actual ambient illumination information. The illumination image is generated based on the document image containing illumination information. The specific implementation process is described below.

[0079] Understandably, to expand the number of training samples, multiple different illumination images can be superimposed on each document image sample that lacks illumination information, thereby generating multiple different superimposed images for each document image sample. For example, if there are 1,000 document image samples and 50 illumination images, after image superposition, 50,000 superimposed images can be generated for model training, significantly expanding the number of model training samples.

[0080] In another embodiment, if the sizes of the illumination image and the document image sample are inconsistent, an image transformation may be performed on the illumination image and / or the document image sample to adjust the sizes of the two images to be the same.

[0081] Step 103: Use the superimposed image and the illumination image as a training sample pair for model training.

[0082] The training sample pairs obtained in step 103 can be used for model training. The trained model can eliminate the lighting information in the input document image and output the document image without lighting information, thereby achieving image enhancement of the document image.

[0083] In this embodiment, a large number of superimposed images can be obtained as training samples by combining and superimposing different lighting images with document image samples that do not contain lighting information. Since the superimposed images are obtained by superimposing the lighting images and the document image samples, the image information of the superimposed images, including size, lighting position, etc., can be determined by the lighting images and the document image samples. Therefore, there is no need for manual shooting and image collection, no need for image registration and lighting information labeling, a high degree of automation, a small amount of calculation, and very little need for manual labeling, which can save a lot of labor costs.

[0084] In this embodiment, the illumination image is separated from the document image containing real illumination information through algorithm processing rather than simulation, and is used as the basis for generating training samples. The authenticity and accuracy of the training samples are very high, which can improve the accuracy of model training.

[0085] In another embodiment, to further expand the number of training samples, the document image obtained by converting the electronic document can be subjected to geometric spatial transformations. For example, the document image can be segmented into multiple sub-images, and the sub-images belonging to different document images can be spliced ​​together to form a new document image as the basis for generating training samples. Alternatively, sub-images of the original document image can be rotated and then spliced ​​together to form a new document image as the basis for generating training samples. Thus, by simply transforming the document image spatially and then combining and superimposing it with images of different illumination, a large amount of sample data that can be used for training can be obtained.

[0086] The following describes the process of generating the illumination image.

[0087] Figure 2 A document image processing method according to an exemplary embodiment of the present disclosure includes the following steps:

[0088] Step 201: Obtain the document image to be processed.

[0089] The document image to be processed may be an image captured of a target object such as a desktop, cloth, billboard, or paper document containing characters. The document image to be processed may also be an image captured by others and obtained from the Internet.

[0090] The target object is the basis for generating the document image to be processed. In order to accurately extract the real lighting information from the document image to be processed, it is best to use a target object with a large difference between the character color and the background color. For example, a paper document with black text on white paper can be used. In this way, the background area of ​​the document image to be processed obtained by photographing the paper document will be almost white, and the color of the character area will be black. Figure 3a Taking the document image to be processed obtained by photographing a paper document as an example, the area where the black fonts and punctuation marks are located in the figure is the character area, and the white part is the background area. The pixel values ​​of the two are quite different, which is conducive to the subsequent generation of the illumination image.

[0091] It is understood that the document image to be processed is obtained by photographing the target object in a lighting environment. The document image to be processed not only includes the original information of the target object, but also contains lighting information. The light source providing the lighting environment can be, but is not limited to, sunlight, incandescent lamps, colored LEDs (light-emitting diodes), etc. From a document image to be processed, a lighting image that reflects the lighting information can be extracted. To increase the number and diversity of training samples, multiple types of target objects can be photographed. The target objects can be placed in different lighting environments and photographed from different angles and distances to obtain multiple document images to be processed.

[0092] Step 202: Determine the initial character area of ​​the characters in the document image to be processed.

[0093] In one embodiment, to determine the initial character region, the document image to be processed can be first binarized to obtain a binary image. The pixel values ​​of the pixels in the binary image are either 0 or 255. The initial character region of the characters can be identified based on the pixel values. The binary image presents a distinct black and white effect, which facilitates the subsequent generation of the illumination image.

[0094] In another embodiment, to determine the initial character area, the area where the characters are located in the document image to be processed can be determined first, and a text detection algorithm can be used but is not limited to be used. The area where the characters are located is then binarized, and the binarized area is determined as the initial character area.

[0095] See also Figure 3a The area in the dotted box in the figure is the initial character area. The area in the dotted box includes not only the characters but also part of the background area. The following steps need to be performed to refine the boundaries of the initial character area to optimize the initial character area.

[0096] Step 203: Optimize the initial character area, determine the character boundaries, and determine the optimized target character area based on the character boundaries.

[0097] The initial character region in the binary image may be optimized using, but is not limited to, a morphological algorithm to determine the boundaries of the characters, thereby accurately determining the character region (target character region) in the image.

[0098] Morphological processing is performed on the initial character area using either morphological opening or morphological closing. Morphological opening involves first eroding the image and then dilating it. This can be used to eliminate small objects, separate objects at fine points, and smooth the boundaries of larger objects without significantly changing their area. Morphological closing involves first dilating the image and then eroding it. Closing can eliminate small black holes (black areas). Users can choose to use either morphological opening or morphological closing based on the characteristics of the characters in the document image.

[0099] The following uses the morphological opening operation as an example to illustrate the specific implementation process of morphological processing of the initial character area:

[0100] Mathematically speaking, the dilation or erosion operation is to convolve the initial character area with the convolution kernel to Figure 3b Taking the initial character area shown as an example, each small square in the figure represents a pixel. The square filled with diagonal lines represents the character area, the white square represents the background area, and the 3*3 dashed square area represents the convolution kernel. The convolution kernel can be of any shape and size, not limited to the 3*3 square composition shown in the figure. The convolution kernel has a separately defined reference point, see the location of the "☆" area in the figure.

[0101] Erosion is an operation to find the local minimum: use the convolution kernel to add the local minimum value in the initial character area ( Figure 3b ), calculates the minimum value of the pixels in the area covered by the convolution kernel each time it moves, and assigns this minimum value to the pixel where the reference point is located. Figure 3c , for Figure 3b A schematic diagram of the results after corrosion treatment, compared Figure 3b and Figure 3c It can be seen that after the corrosion operation, the diagonal filling area is reduced.

[0102] Dilation is an operation to find the local maximum: using the convolution kernel Figure 3c The convolution kernel moves in the image, and each time it moves, the maximum value of the pixels in the area covered by the convolution kernel is calculated, and this maximum value is assigned to the pixel where the reference point is located, so that the diagonal filling area in the image will gradually grow. Figure 3d , for Figure 3c A schematic diagram of the result after the expansion process. The diagonal filled area in the figure is the optimized character area, and its boundary is also the boundary of the character area.

[0103] Step 204: Eliminate the target character region from the document image to be processed, and generate a lighting image based on the document image to be processed with the target character region eliminated.

[0104] Among them, the illumination image can reflect the illumination information of the environment.

[0105] Figure 3e This is an exemplary embodiment of the present disclosure showing a document image to be processed after the character area is removed. After the text area of ​​the document image to be processed is removed, only the lighting information remains. The dot-filled area in the figure represents the area containing lighting information, and the white area in the figure represents the blank area after the target character area is removed. Since the lighting information of the area is also removed when the character area in the image is removed, some lighting information is missing in the image at this time and cannot fully reflect the actual lighting information. It is necessary to fill the pixels in the blank area with lighting information, which can be done by, but not limited to, the following methods:

[0106] For each pixel in the blank area, the pixel values ​​of adjacent pixels within a preset range from the pixel are fitted using an interpolation algorithm, and the pixel value is filled in at the pixel point. The preset range of pixels can be set according to actual needs.

[0107] by Figure 3e Taking the image shown as an example, assuming that the preset range is one adjacent pixel point, when pixel point a is filled, the adjacent pixel points of pixel point a are pixel points b1 to b8.

[0108] In one embodiment, the pixel values ​​of adjacent pixels, i.e., pixels b1 to b8, may be averaged, and the average value obtained may be assigned to pixel a. Figure 3e Since pixels b4, b5, and b6 are blank pixels, their pixel values ​​can be assigned to 0 and then the average is calculated, or the average is calculated based only on the pixel values ​​of pixels b1, b2, b3, b7, and b8.

[0109] In another embodiment, the nearest neighbor interpolation method can be used. Figure 3e For example, among the adjacent pixels of pixel a, pixels b1, b2, b3, b7, and b8 have illumination information. A pixel can be selected from these four pixels and its pixel value is assigned to pixel a. This algorithm has a low computational complexity.

[0110] For each pixel point in the blank area, the pixel value is filled in the above manner, so that each pixel point in the image includes lighting information, thereby obtaining a lighting image with good visual perception and including real lighting information.

[0111] In the disclosed embodiment, the boundaries of the character areas in the document image can be accurately located, and an illumination image containing illumination information can be generated based on the document image after the character areas are removed along the boundaries, which serves as the basis for generating training samples, thereby solving the current problem of difficulty in obtaining illumination information.

[0112] Figure 4 This is another document image processing method according to an exemplary embodiment of the present disclosure. In this embodiment, a document image to be processed obtained by photographing a paper document is taken as an example to illustrate the specific implementation process of the document image processing method. Figure 4 , the method comprises the following steps:

[0113] Step 401: Acquire a document image to be processed obtained by photographing a paper document.

[0114] The paper document includes a character area and a background area. In this step, the paper document is the basis for generating the document image. In order to accurately extract the true lighting information from the document image to be processed, it is best to use a paper document with a large difference between the character color and the background color. Preferably, a paper document with black text on white paper is used. In this way, the color of the background area of ​​the document image to be processed obtained by photographing the paper document is almost white, and the color of the character area is black. Figure 3a Taking the document image obtained by photographing a paper document as an example, the area where the black fonts and punctuation marks are located in the figure is the character area, and the white part is the background area. The pixel values ​​of the two are quite different, which is conducive to the subsequent generation of the illumination image.

[0115] It can be understood that the document image to be processed is obtained by photographing a paper document in a lighting environment. The document image to be processed includes not only the original information of the paper document, but also lighting information. The light source providing the lighting environment can be, but is not limited to, sunlight, incandescent lamps, colored LEDs (light-emitting diodes), etc. A lighting image including lighting information can be extracted from a document image to be processed. To increase the number and diversity of training samples, multiple paper documents can be obtained, each of which can be placed in different lighting environments and photographed from different angles and distances to obtain multiple document images, i.e., multiple document images to be processed.

[0116] Step 402: Determine the initial character area in the document image to be processed.

[0117] Among them, the initial character region in the image can be determined by using, but not limited to, a text detection algorithm. The text detection algorithm can only identify an initial character region of the characters in the image. Figure 3a The area framed by dotted lines in the figure is the initial character area, which includes not only the characters but also part of the background area. The following steps need to be performed to identify the accurate boundaries of the character area and optimize the character area.

[0118] Step 403: Optimize the initial character region based on a morphological algorithm, determine the character boundaries, and determine the optimized target character region based on the character boundaries.

[0119] In order to facilitate morphological processing, before morphological processing, the initial character area needs to be binarized and the pixel values ​​in the character area are assigned to 0 or 255, so that the initial character area presents an obvious black and white effect.

[0120] In another embodiment, the document image to be processed may be binarized to obtain a binary image, and then the initial character area may be identified from the binary image. The pixel values ​​of the pixels in the initial character area have only two values: 0 and 255.

[0121] Morphological processing can be performed on the initial character area using either morphological opening or morphological closing. Morphological opening involves first eroding the image and then dilating it. This can be used to eliminate small objects, separate objects at fine points, and smooth the boundaries of larger objects without significantly changing their area. Morphological closing involves first dilating the image and then eroding it. This operation can eliminate small black holes (black areas). Users can choose to use either morphological opening or morphological closing based on the characteristics of the characters in the document image.

[0122] The following uses the morphological opening operation as an example to illustrate the specific implementation process of morphological processing of the initial character area:

[0123] Mathematically speaking, the dilation or erosion operation is to convolve the initial character area with the convolution kernel to Figure 3b Taking the initial character area shown as an example, each small square in the figure represents a pixel. The square filled with diagonal lines represents the character area, the white square represents the background area, and the 3*3 dashed square area represents the convolution kernel. The convolution kernel can be of any shape and size, not limited to the 3*3 square composition shown in the figure. The convolution kernel has a separately defined reference point, see the location of the "☆" area in the figure.

[0124] Erosion is an operation to find the local minimum: use the convolution kernel to add the local minimum value in the initial character area ( Figure 3b), calculates the minimum value of the pixels in the area covered by the convolution kernel each time it moves, and assigns this minimum value to the pixel where the reference point is located. Figure 3c , for Figure 3b A schematic diagram of the results after corrosion treatment, compared Figure 3b and Figure 3c It can be seen that after the corrosion operation, the diagonal filling area is reduced.

[0125] Dilation is an operation to find the local maximum: using the convolution kernel Figure 3c The convolution kernel moves in the image, and each time it moves, the maximum value of the pixels in the area covered by the convolution kernel is calculated, and this maximum value is assigned to the pixel where the reference point is located, so that the diagonal filling area in the image will gradually grow. Figure 3d , for Figure 3c A schematic diagram of the result after the expansion process. The diagonal filled area in the figure is the optimized character area, and its boundary is also the boundary of the character area.

[0126] Step 404: Eliminate the target character region in the document image to be processed along the boundary of the characters, and generate a lighting image based on the document image to be processed with the target character region eliminated.

[0127] Figure 3e This is an exemplary embodiment of the present disclosure showing a document image to be processed after the target character area is removed. After the text area of ​​the document image to be processed is removed, only the lighting information remains. The dot-filled area in the figure represents the area including the lighting information, and the white area in the figure represents the blank area after the character area is removed. Since the lighting information of the area is also removed when the character area in the image is removed, some lighting information is missing in the image at this time and cannot fully reflect the actual lighting information. It is necessary to fill the pixels in the blank area with lighting information, which can be done by, but not limited to, the following methods:

[0128] For each pixel in the blank area, the pixel values ​​of adjacent pixels within a preset range from the pixel are fitted using an interpolation algorithm, and the pixel value is filled in at the pixel point. The preset range of pixels can be set according to actual needs.

[0129] by Figure 3e Taking the document image as an example, assuming that the preset range is 1 adjacent pixel point, when pixel point a is filled, the adjacent pixel points of pixel point a are pixel points b1 to b8.

[0130] In one embodiment, the pixel values ​​of adjacent pixels, i.e., pixels b1 to b8, may be averaged, and the average value obtained may be assigned to pixel a. Figure 3eSince pixels b4, b5, and b6 are blank pixels, their pixel values ​​can be assigned to 0 and then the average is calculated, or the average is calculated based only on the pixel values ​​of pixels b1, b2, b3, b7, and b8.

[0131] In another embodiment, the nearest neighbor interpolation method can be used. Figure 3e For example, among the adjacent pixels of pixel a, pixels b1, b2, b3, b7, and b8 have illumination information. A pixel can be selected from these four pixels and its pixel value is assigned to pixel a. This algorithm has a low computational complexity.

[0132] For each pixel point in the blank area, the pixel value is filled in the above manner, so that each pixel point in the document image includes lighting information, thereby obtaining a lighting image with good visual perception and including real lighting information.

[0133] In the disclosed embodiment, the boundaries of the character areas in the document image can be accurately located, and an illumination image containing real illumination information can be generated based on the document image after the character areas are removed along the boundaries, which serves as the basis for generating training samples, thereby solving the current problem of difficulty in obtaining illumination information.

[0134] Corresponding to the aforementioned method embodiments, the present disclosure also provides embodiments of an apparatus and equipment used therein.

[0135] Figure 5 1 is a module diagram of a document image processing device according to an exemplary embodiment of the present disclosure. The processing device includes: a determination module 51 , an optimization module 52 and a generation module 53 .

[0136] A determination module 51 is used to determine an initial character area where characters are located in the document image to be processed;

[0137] An optimization module 52 is configured to optimize the initial character region, determine the boundaries of the characters, and determine an optimized target character region based on the boundaries of the characters;

[0138] The generating module 53 is configured to remove the target character region from the document image to be processed, and generate a lighting image based on the document image to be processed from which the target character region has been removed, wherein the lighting image is configured to reflect lighting information of the environment.

[0139] Optionally, when optimizing the initial character area, the optimization module is configured to:

[0140] performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image;

[0141] Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

[0142] Optionally, when generating the illumination image based on the document image to be processed from which the target character area is removed, the generating module is configured to:

[0143] For each pixel point in the blank area, the pixel values ​​of adjacent pixels within a preset range from the pixel point are fitted based on an interpolation algorithm, and the pixel values ​​are filled at the pixel point. The blank area is the area of ​​the document image to be processed from which the target character area is removed.

[0144] The document image to be processed that has been filled with pixel values ​​is determined as the illumination image.

[0145] Figure 6 6 is a module diagram of another document image processing device according to an exemplary embodiment of the present disclosure. The processing device includes: an acquisition module 61, a determination module 62, an optimization module 63 and a generation module 64.

[0146] The acquisition module 61 is used to acquire the document image to be processed.

[0147] Among them, the acquisition module can obtain the image of the document to be processed from this device, and the image of the document to be processed is obtained by this device photographing a paper document including characters; the acquisition module can also obtain the image of the document to be processed photographed by other devices, or obtain the image of the document to be processed from the network.

[0148] a determination module 62 for performing binarization processing on the document image to be processed to obtain a binary image, and determining an initial character region in the document image from the binary image;

[0149] an optimization module 63 for performing morphological processing on the initial character region to determine the boundaries of the character and determining an optimized target character region based on the boundaries of the character;

[0150] The generating module 64 is configured to remove a target character region from the document image to be processed, and generate a lighting image based on the document image to be processed from which the target character region has been removed, wherein the lighting image is configured to reflect lighting information of the environment.

[0151] Optionally, when performing morphological processing on the initial character area, the optimization module is configured to:

[0152] performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image;

[0153] Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

[0154] Optionally, when generating the illumination image based on the document image to be processed from which the target character area is removed, the generating module is configured to:

[0155] For each pixel point in the blank area, fitting the pixel values ​​of adjacent pixels within a preset range from the pixel point based on an interpolation algorithm, and filling the pixel point with the pixel value, wherein the blank area is the area of ​​the document image to be processed from which the optimized character area is removed;

[0156] The document image filled with pixel values ​​is determined as the illumination image.

[0157] Figure 7 A training sample generation device according to an exemplary embodiment of the present disclosure is shown, and the generation device includes: an acquisition module 71 , a superposition module 72 and a sample generation module 73 .

[0158] An acquisition module 71 is used to acquire a document image sample;

[0159] an overlay module 72 for overlaying a light image on the document image sample to obtain an overlay image, wherein the light image is obtained by any of the above-mentioned document image processing devices;

[0160] The sample generation module 73 is configured to use the superimposed image and the illumination image as a training sample pair for training a model, wherein the model is configured to eliminate illumination information in an input document image.

[0161] Optionally, the document image not including illumination information includes:

[0162] a first document image converted from an electronic document;

[0163] and / or, performing a spatial geometric transformation on the first document image to obtain a second document image.

[0164] Regarding the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the relevant methods and will not be elaborated on here.

[0165] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative work.

[0166] Figure 8 The present invention is a block diagram of a device for document image processing according to an embodiment of the present invention. The device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0167] like Figure 8 As shown, the device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816. The device also includes an antenna module (for example, which can be connected to the communication component 816), which includes: a radiator, a grounding point, a feeding end, and a resonant circuit. The radiator includes an open end, and the grounding point is determined on the radiator. The feeding end is electrically connected to a first connection point on the radiator. The first end of the resonant circuit is electrically connected to the first connection point, the second end of the resonant circuit is grounded, and the resonant circuit includes an adjustable unit. The distance from the first connection point to the open end is less than the distance from the first connection point to the grounding point.

[0168] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0169] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0170] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0171] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0172] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0173] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0174] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800. The sensor assembly 814 can also detect changes in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and temperature changes of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0175] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0176] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the method described in any of the above embodiments.

[0177] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0178] Figure 9 FIG. 1 is a block diagram of another document image processing device 900 according to an exemplary embodiment. For example, the device 900 may be provided as a server. Figure 9 The device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as applications, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the method described in any of the above embodiments.

[0179] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0180] An embodiment of the present disclosure further provides a training sample generation device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the training sample generation method described in any of the above embodiments when executing the instructions.

[0181] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0182] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for processing a document image, characterized in that: The processing method comprises: Determining an initial character area where characters are located in a document image to be processed, wherein the document image to be processed includes character information and lighting information; Optimizing the initial character region, determining the boundaries of the characters, and determining an optimized target character region based on the boundaries of the characters; After removing the target character area from the document image to be processed, for each pixel point in the blank area, fitting the pixel values ​​of adjacent pixels within a preset range from the pixel point based on an interpolation algorithm, and filling the pixel point with the pixel value, the blank area being the area in the document image to be processed from which the target character area is removed; The document image to be processed that has been filled with pixel values ​​is determined as a lighting image, where the lighting image is used to reflect lighting information of an environment.

2. The document image processing method according to claim 1, wherein: Determining an initial character area in the document image to be processed includes: performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image; Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

3. The document image processing method according to claim 1 or 2, wherein: Optimizing the initial character area includes: The initial character region is optimized based on a morphological algorithm.

4. The document image processing method according to claim 1, wherein: The difference between the pixel value of the background area in the to-be-processed document image and the pixel value of the character area in the to-be-processed document image is greater than a pixel threshold.

5. A method for generating training samples, characterized in that: The generation method comprises: Obtain document image samples; superimposing a light image on the document image sample to obtain a superimposed image, wherein the light image is obtained by the document image processing method according to any one of claims 1 to 4; The superimposed image and the document image sample are used as a training sample pair for model training, and the model is used to eliminate lighting information in an input document image.

6. The method for generating training samples according to claim 5, wherein: The document image sample includes at least one of the following: a first document image converted from an electronic document; A second document image is obtained by performing spatial geometric transformation on the first document image.

7. A document image processing device, characterized in that: The processing device comprises: a determination module, configured to determine an initial character region where characters are located in a document image to be processed, wherein the document image to be processed includes character information and lighting information; an optimization module, configured to optimize the initial character region, determine the boundaries of the characters, and determine an optimized target character region based on the boundaries of the characters; A generation module is used to remove the target character area from the document image to be processed, and for each pixel point in the blank area, fit the pixel values ​​of adjacent pixel points within a preset range from the pixel point based on an interpolation algorithm, and fill the pixel value at the pixel point, and then determine the document image to be processed filled with pixel values ​​as an illumination image, the blank area is the area in the document image to be processed from which the target character area is removed, and the illumination image is used to reflect the lighting information of the environment.

8. The document image processing device according to claim 7, wherein: When optimizing the initial character area, the optimization module is used to: performing binarization processing on the document image to be processed to obtain a binary image, and identifying the initial character area from the binary image; Alternatively, the region where the characters are located in the document image to be processed is binarized, and the binarized region is determined as the initial character region.

9. A device for generating training samples, characterized in that: The generating device comprises: An acquisition module, used for acquiring document image samples; a superimposition module, configured to superimpose a light image on the document image sample to obtain a superimposed image, wherein the light image is obtained by the document image processing device according to any one of claims 7 to 8; The sample generation module is used to use the superimposed image and the document image sample as a training sample pair for model training, and the model is used to eliminate lighting information in the input document image.

10. The training sample generating device according to claim 9, characterized in that: The document image sample includes at least one of: a first document image converted from an electronic document; A second document image is obtained by performing spatial geometric transformation on the first document image.

11. A document image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of the document image processing method according to any one of claims 1 to 4 when executing the execution instruction.

12. A training sample generation device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of the method for generating training samples according to claim 5 or 6 when executing the execution instruction.

Images