Document image region separation method, device and storage medium

Abstract

The present invention provides a method, device and storage medium for region separation of document images. A binary image is obtained by preprocessing the document image, and a connected region analysis is performed on the binary image to obtain a first connected region set. According to the attribute characteristics of each connected region in the first connected region set, the first connected region set is divided into a second connected region set and a third connected region set, wherein the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements. Further, according to the attribute characteristics and adjacent position information of each connected region in the third connected region set, the connected regions of text elements are determined from the third connected region set. The above method can extract more non-text elements from the text region, thereby improving the accuracy of text-non-text separation of the document image.

Description

Technical Field

[0001] The present invention relates to the technical field of layout analysis, and in particular to a method, device and storage medium for region separation of a document image. Background Art

[0002] With the promotion and popularization of electronic reading materials such as electronic journals and electronic books, more and more traditional paper reading materials are converted into digital images through image acquisition devices such as image scanners and cameras, and then shared on the Internet. We call this kind of digital image that mainly carries text information such as text, pictures, and tables document images. Document images are derived from paper files and have the characteristics of intuitive content, easy to carry and transmit, etc., so they are widely used in various industries.

[0003] The layout analysis of document images mainly uses computers to automatically process and segment the layout data of document images, and identify the location and attributes of text, pictures, graphics, tables and other areas on the image layout. At present, there are two main methods for regional separation of document images: one is a pixel-by-pixel determination method, which mainly targets each pixel in the image, calculates the edge strength value of each pixel according to the gray value of the pixel, and then determines the binary image corresponding to the image to be processed according to the edge strength value of each pixel, determines the text area determination value of each pixel according to the binary image and the gray value of each pixel, and finally determines the text area and non-text area of ​​the image according to the determination value of each pixel. The other is a text detection technology based on deep learning, which uses a deep learning network to detect text in the image, such as Faster RCNN, EAST, Textbox++ and other text detection methods, and then uses the detected text as the text area.

[0004] The first method is based on the calculation of a single pixel and the surrounding pixel values, which can only focus on the local part of the image, but cannot grasp the overall structure of the image and the position of each element. The accuracy and robustness of regional separation are low. The second method can only detect text, and cannot distinguish the text in pictures and tables from the paragraph text in normal documents. Summary of the invention

[0005] The present invention provides a method, a device and a storage medium for region separation of a document image, which improves the accuracy of text and non-text separation of a document image.

[0006] A first aspect of the present invention provides a method for region separation of a document image, comprising:

[0007] Preprocess the document image to obtain a binary image;

[0008] Performing a connected region analysis on the binary image to obtain a first connected region set;

[0009] Acquire and divide the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set; the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements;

[0010] The connected regions of the text elements are determined from the third connected region set by acquiring and determining the connected regions according to the attribute characteristics and adjacent position information of each connected region in the third connected region set.

[0011] Optionally, the attribute features of each of the connected regions include the number of pixels, pixel density, aspect ratio of each of the connected regions, and the first number of other connected regions nested in the minimum circumscribed rectangle of each of the connected regions.

[0012] In a possible implementation, dividing the first connected region set into a second connected region set and a third connected region set according to attribute characteristics of each connected region in the first connected region set includes:

[0013] It is determined whether the attribute characteristics of each connected region in the first connected region set meet the first non-text element condition, and the connected regions whose attribute characteristics meet the first non-text element condition are divided into the second connected region set.

[0014] In a possible implementation, the first non-text element condition includes at least one of the following conditions:

[0015] The number of pixels in the connected area is less than the preset number of pixels;

[0016] The pixel density of the connected area is less than the preset pixel density;

[0017] The aspect ratio of the connected area is smaller than the preset aspect ratio;

[0018] A first number of other connected regions nested in the minimum circumscribed rectangle of the connected region is greater than a first value.

[0019] In a possible implementation manner, obtaining adjacent position information of each connected region in the third connected region set includes:

[0020] Performing blank area analysis on each connected area in the third connected area set to determine adjacent connected areas corresponding to each connected area in the third connected area set;

[0021] Obtain adjacent position information corresponding to each connected area in the third connected area set, wherein the adjacent position information includes a distance between each connected area and an adjacent connected area corresponding to each connected area, and the number of left connected areas and the number of right connected areas of each connected area.

[0022] In a possible implementation, determining the connected regions of the text element from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set includes:

[0023] Determining whether the attribute feature of each connected region in the third connected region set meets the second non-text element condition;

[0024] Determining whether adjacent position information of each connected region in the third connected region set satisfies a third non-text element condition;

[0025] The connected regions in the third connected region set that do not satisfy the second non-text element condition and the connected regions that satisfy the second non-text element condition but do not satisfy the third non-text element condition are determined as connected regions of text elements.

[0026] In a possible implementation, the second non-text condition includes:

[0027] The first and second conditions; or,

[0028] The first condition and the third condition; or,

[0029] the first condition, the second condition and the third condition;

[0030] Among them, the first condition is A i =max(Ω1)∩A i >k1×median(Ω1); the second condition is H i =max(Ω2)∩H i >k2×median(Ω2); the third condition is W i =max(Ω3)∩W i >k3×median(Ω3);

[0031] In the formula, Ω1 represents the set of pixel numbers of each connected region in the third connected region set, Ω2 represents the set of heights of each connected region in the third connected region set, Ω3 represents the set of widths of each connected region in the third connected region set, mean represents the calculation of the mean, median represents the calculation of the median, A i represents the number of pixels of the i-th connected region in the third connected region set, Hi represents the height of the minimum circumscribed rectangle of the i-th connected region in the third connected region set, W i Represents the width of the minimum circumscribed rectangle of the ith connected region in the third connected region set.

[0032] In a possible implementation, the third non-text element condition includes:

[0033] The distance between each connected area and the adjacent connected area corresponding to each connected area is greater than or equal to a preset distance; and / or

[0034] A second number of adjacent connected regions corresponding to each connected region is greater than or equal to a second value, and the second number is a maximum value between the number of left connected regions and the number of right connected regions.

[0035] In a possible implementation, the method further includes:

[0036] Obtaining the first pixel number of the minimum circumscribed rectangle of the connected area overlapping with the connected area of ​​the non-text element;

[0037] Obtaining a second number of pixels after the connected area of ​​the non-text element is dilated;

[0038] If the first number of pixels is greater than the second number of pixels, the connected area overlapping with the connected area of ​​the non-text element is determined as the connected area of ​​the text element.

[0039] A second aspect of the present invention provides a document image region separation device, comprising:

[0040] An image preprocessing module is used to preprocess the document image to obtain a binary image;

[0041] A connected region analysis module, used for performing connected region analysis on the binary image to obtain a first connected region set;

[0042] An acquisition module, used for acquiring attribute characteristics of each connected region in the first connected region set;

[0043] A connected region division module, used for dividing the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set; the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements;

[0044] The acquisition module is further used to acquire attribute characteristics and adjacent position information of each connected area in the third connected area set;

[0045] The connected region division module is further used to determine the connected regions of the text elements from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set.

[0046] A third aspect of the present invention provides a device for separating regions of a document image, comprising:

[0047] Memory;

[0048] Processor; and

[0049] Computer programs;

[0050] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in any one of the first aspects of the present invention.

[0051] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in any one of the first aspects of the present invention.

[0052] The embodiment of the present invention provides a method, device and storage medium for region separation of a document image. A binary image is obtained by preprocessing the document image, and a connected region analysis is performed on the binary image to obtain a first connected region set. According to the attribute characteristics of each connected region in the first connected region set, the first connected region set is divided into a second connected region set and a third connected region set, wherein the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements. Further, according to the attribute characteristics and adjacent position information of each connected region in the third connected region set, the connected regions of text elements are determined from the third connected region set. The above method can extract more non-text elements from the text region, thereby improving the accuracy of text-non-text separation of the document image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a method for region separation of a document image provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of pixel adjacency relationship in an image provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the connected region analysis provided by an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a connected area provided in an embodiment of the present invention;

[0057] Figure 5A schematic diagram of the positional relationship between two connected areas provided in an embodiment of the present invention;

[0058] Figure 6 A schematic diagram of blank area analysis of a Chinese document provided by an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of text and non-text separation results of the document image region separation method provided by an embodiment of the present invention;

[0060] Figure 8 A schematic flow chart of a method for region separation of a document image provided by another embodiment of the present invention;

[0061] Fig. 9 A functional structural diagram of a device for separating regions of document images provided by an embodiment of the present invention;

[0062] Fig.10 A schematic diagram of the hardware structure of a document image region separation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0064] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0065] The "one embodiment" or "another embodiment" mentioned throughout the specification of the present invention means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in some embodiments" or "in this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0066] The document image region separation method provided by the embodiment of the present invention processes the connected regions in the document image as units, separates the text and non-text regions in the document image based on the attribute characteristics of the connected regions and the positional relationship between the connected regions, and can effectively separate regions for various document layouts and documents in different languages, with high separation accuracy and stable effect. In addition, the document image region separation method provided by the embodiment can not only distinguish between text regions and non-text regions, but also further eliminate more non-text elements from the connected regions that overlap with the connected regions of non-text elements, and finally obtain accurate text regions.

[0067] The document image region separation method provided by the present invention is described in detail below in conjunction with several specific embodiments. The following specific embodiments can be combined with each other, and the same or similar contents will not be repeated in different embodiments.

[0068] Figure 1 The flowchart of the document image region separation method provided by one embodiment of the present invention is shown in FIG. The execution subject of the method can be executed by any device for executing document image region separation, which can be implemented by software and / or hardware, and can be a camera or an intelligent analysis server. Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0069] Step 101: pre-process the document image to obtain a binary image.

[0070] In this embodiment, the document image can be a photographed document or a scanned document, which is not limited in this embodiment. Specifically, the document image includes a text area and a non-text area, the text area includes text elements such as Chinese characters, English letters, numbers, symbols, formulas, etc., and the non-text area includes pictures, table pictures, icons, etc.

[0071] After the document image is acquired, the document image is preprocessed, and the preprocessing process includes steps such as image binarization, rotation correction, and perspective correction.

[0072] Among them, image binarization is the process of setting the grayscale value of the pixel points on the document image to 0 or 255, that is, the whole image presents an obvious black and white effect. Image binarization greatly reduces the amount of data in the image and improves the speed of image processing.

[0073] In planar image processing, due to the lens angle, or the scanner's failure to scan horizontally and vertically according to its rows and columns during image scanning, it is easy to cause the photographed or scanned document image to appear tilted or deformed. Therefore, it is necessary to perform rotation correction and perspective correction on the document image to ensure that the document image to be processed does not have deformations such as rotation and bending, for example, the rotation angle is less than 2 degrees.

[0074] In this embodiment, the order of the steps of preprocessing the document image is not limited. The document image may be binarized first, and then rotation correction and perspective correction may be performed on the binarized image; or rotation correction and perspective correction may be performed on the document image first, and then the corrected document image may be binarized.

[0075] Step 102: Perform connected region analysis on the binary image to obtain a first connected region set.

[0076] Connected region analysis is the process of extracting and marking connected regions in a binary image. A connected region refers to an image region composed of foreground pixels with the same pixel value and adjacent positions.

[0077] Figure 2 A schematic diagram of the pixel adjacency relationship in an image provided by an embodiment of the present invention, such as Figure 2 As shown, there are two common adjacency relationships of image pixels: 4-adjacency and 8-adjacency. This embodiment uses 8-adjacency to determine whether pixels belong to the same connected region.

[0078] This embodiment uses the skimage library in Python to mark the connected areas in the image. Figure 3 A schematic diagram of the connected region analysis provided by an embodiment of the present invention. Figure 3 As shown, each connected area corresponds to an external rectangular box.

[0079] The i-th connected region is denoted as CC i , the connected regions have the following properties:

[0080] (1)B(CC i ) indicates CC i The minimum circumscribed rectangle of a coordinate system is parallel to the coordinate axes. i ,Yl i ) and (Xr i ,Yr i ) are the coordinates of the upper left corner and lower right corner of the rectangular box respectively.

[0081] (2)A i Indicates CC i The number of pixels in .

[0082] (3)A iB Indicates B(CC i ) size, W i and H i Respectively represent B(CC i )’s width and height;

[0083] (4)λ i Indicates CC i The density of i =A i / A i B .

[0084] (5)Inc i Indicates nesting in B(CC i ) in other rectangular boxes B(CC j ), j≠i (Note: Calculate Inc i Before, press Yl on the rectangular frame i Component sorting can greatly reduce the computational complexity).

[0085] (6)HW i rate Indicates CC i The aspect ratio is calculated as HW i rate =min(W i ,H i ) / max(W i ,H i ).

[0086] It should be noted that, the first connected region set is obtained by performing connected region analysis on the binary image. The first connected region set includes multiple connected regions, and the multiple connected regions include text regions and non-text regions. It is necessary to identify and divide the multiple connected regions in the first connected region set according to steps 103 to 106 to finally obtain accurate text regions. The specific implementation process is shown below.

[0087] Step 103: Obtain attribute features of each connected region in the first connected region set.

[0088] As can be seen from step 102 , the attribute characteristics of each connected region in the first connected region set include the number of pixels, pixel density, aspect ratio, and the first number of other connected regions nested in the minimum circumscribed rectangle of each connected region.

[0089] Step 104: divide the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set.

[0090] The second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements.

[0091] Specifically, it is determined whether the attribute characteristics of each connected area in the first connected area set meet the first non-text element condition, and the connected areas whose attribute characteristics meet the first non-text element condition are divided into the second connected area set. The first non-text element condition is used to perform preliminary filtering on non-text elements in the document image. If the connected area meets the first non-text element condition, it is regarded as a connected area of ​​the non-text element.

[0092] The first non-text element condition includes at least one of the following conditions:

[0093] (1) The number of pixels in the connected area is less than the preset number of pixels. Specifically, it can be expressed by formula A i <T area Indicates that T area The preset number of pixels is usually set at 5-7 pixels. For example, T area =6pixel, indicating that connected areas with very small pixel values ​​are considered non-text.

[0094] (2) The pixel density of the connected area is less than the preset pixel density. Specifically, it can be expressed by the formula λ i <T dens Indicates that T dens is the preset pixel density, usually set in [0.05,0.07], for example, T dens =0.06, indicating that the pixel density of the connected area is too low, which may be non-text such as noise, diagonal elements, rectangular boxes, etc.

[0095] (3) The aspect ratio of the connected area is smaller than the preset aspect ratio.

[0096] Specifically, in H i <W i (the height of the connected region is less than its width), HW i rate <T rate , where T1 rate is the first preset aspect ratio (the preset aspect ratio for horizontal filtering), usually set in [0.05, 0.07], for example T1 rate = 0.06. Generally speaking, the width of a text pixel cannot be greater than 16.66 times its height.

[0097] In H i >W i (the height of the connected region is greater than its width), HW i rate <T rate , where T2rate The second preset aspect ratio (the preset aspect ratio for vertical filtering), usually set in [0.02, 0.04], for example T2 rate =0.03.

[0098] Usually, the preset width and height for vertical filtering are stricter than those for horizontal filtering.

[0099] (4) The first number of other connected areas nested in the minimum circumscribed rectangle of the connected area is greater than the first value. Specifically, the formula Inc i >T inside Indicates that T inside is the first value, usually set to 4, indicating that if the first number of other connected areas nested in the minimum bounding rectangle of the connected area is greater than 4, they are considered non-text. This condition applies to both Latin and Chinese documents.

[0100] It should be noted that the above preset parameters (including the preset number of pixels, the preset pixel density, the preset aspect ratio, and the first value) are empirical values ​​obtained through a large amount of calculation and verification, and are suitable for documents of various types and resolutions.

[0101] The purpose of this step is to eliminate obvious noise or non-text elements in the binary image and obtain a preliminary separation result. It should be noted that, in addition to the connected regions of text elements, the third connected region set also includes connected regions of non-text elements identified as text elements, that is, the above process may result in non-text elements being misjudged as text elements. In order to improve the accuracy of text-non-text separation, it is necessary to further perform steps 105 and 106 to further separate the non-text elements in the text region. The specific process is described below.

[0102] Step 105: Obtain attribute features and adjacent position relationships of each connected region in the third connected region set.

[0103] As can be seen from step 102, the attribute characteristics of each connected region in the third connected region set include the number of pixels, pixel density, aspect ratio of each connected region, and the first number of other connected regions nested in the minimum circumscribed rectangle of each connected region.

[0104] In this step, the adjacent position relationship of each connected area in the third connected area set is obtained, including the following steps: performing blank area analysis on each connected area in the third connected area set to determine the adjacent connected areas corresponding to each connected area in the third connected area set; and obtaining the adjacent position information corresponding to each connected area in the third connected area set.

[0105] The adjacent position information includes the distance between each connected region and the adjacent connected region corresponding to each connected region, the number of left connected regions and the number of right connected regions of each connected region.

[0106] For ease of understanding, the following Figure 4 as well as Figure 5 The above-mentioned blank area analysis is explained.

[0107] Figure 4 A schematic diagram of a connected area provided in an embodiment of the present invention. Figure 4 , in the above blank area analysis, for the connected area CC i The following variables need to be calculated:

[0108] (1) LNN i and RNN i :LNN i Indicates CC i The left connected region (CC i The left neighbor of i Indicates CC i The right connected region (CC i right neighbor).

[0109] It should be noted that CC i The left connected area or right connected area of ​​LNN may include one or more. i Specifically indicates the closest CC i The left connected region, RNN i Specifically indicates the closest CC i The right connected region of .

[0110] For example, Figure 4 CC3's LNN3={CC2},RNN3={CC4}; CC4's LNN4={CC3},

[0111] It should be pointed out that CC i The left connected area and the right connected area need to be CC i In the same row direction.

[0112] (2)LNWS i , RNWS i :LNWS i Indicates CC i The distance between the left connected area (blank size), RNWS i Indicates CC i The distance between the right connected region (blank size). Let CC iThe left neighbor is LNN i ={CC j}, the right neighbor is RNN i ={CC k}, so LNWS i and RNWS i The calculation formula is:

[0113] LNWS i =Xl i -Xr j

[0114] RNWS i =Xl k -Xr i

[0115] (3)LN i , R.N. i :LN i Indicates CC i The set of other connected regions when they become the right connected regions of other connected regions; RN i Indicates CC i The set of other connected regions when it becomes the left connected region of other connected regions.

[0116]

[0117]

[0118] For example, Figure 4 In CC3, LN3={CC1,CC2}, RN3={CC4,CC5}.

[0119] (4)numLN i , numRN i :numLN i Indicates CC i The number of other connected regions when they become the right connected regions of other connected regions; numRN i Indicates CC i The number of other connected regions when it becomes the left connected region of other connected regions.

[0120] numLN i =|LN i |

[0121] numRN i =|RN i |

[0122] For example, Figure 4 In CC3, numLN3 = numRN3 = 2

[0123] (5) WS: Represents the set composed of all blank areas.

[0124] WS = {RNWS i > 0 | CC i ∈ CC u}} or WS = {LNWS i > 0 | CC i ∈ CC u}}

[0125] Wherein, CC u represents all connected areas in the document image.

[0126] By performing blank area analysis on each connected area in the third set of connected areas, the above variables of each connected area in the third set of connected areas are obtained, and the adjacent position information between each connected area and its nearby connected areas is known.

[0127] The above blank area analysis only considers the case where the connected areas in the document image are separated from each other, as Figure 4 shown. However, there may also be cases where the connected areas in the document image overlap. Figure 5 This is a schematic diagram of the positional relationship between two connected areas provided by an embodiment of the present invention, as Figure 5 shown, and there may be cases where two connected areas overlap each other.

[0128] Specifically, for English documents, there may be Figure 5 the overlapping situation of (b), and for Chinese documents, there may be Figure 5 the overlapping situations of (b), (c), and (d). It should be noted that neither English documents nor Chinese documents will have Figure 5 the situation of (e). At this time, CC i and CC j will surely be connected into one connected area.

[0129] Exemplarily, there is Figure 5 the overlapping situation of (b) for italic words in both English and Chinese, and there is Figure 5 the overlapping situation of (c) for the Chinese character "画", and there is Figure 5 the overlapping situation of (d) for the Chinese character "回".

[0130] For Figure 5 the overlapping situation of (b), judge the left - right positional relationship between CC i and CC j . In Figure 5 (b), CC i is the left connected area of CC j , and CC j is CC iThe right connected area of ​​, the distance between them is set to 0; Figure 5 (c) Overlap, CC i and CC j The left / right connected regions are mutually connected, and the distance between them is set to 0; Figure 5 (d) The overlap is nested and suppressed. CC j There is no left / right connected region, CC i and CC j The distance is set to 0.

[0131] It should be pointed out that, when determining the adjacent connected area corresponding to a certain connected area, this embodiment not only considers the positional relationship of the connected areas being separated from each other, but also considers the positional relationship of the connected areas being overlapped left and right, overlapped up and down, and nested, thereby providing a more accurate judgment basis for subsequent conditional judgments, thereby avoiding English letters or Chinese characters in Chinese and English documents from being misjudged as non-text elements.

[0132] Step 106: Determine the connected regions of the text element from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set.

[0133] In this embodiment, the connected regions in the third connected region set may include non-text elements that are not identified in step 104, such as small icons and noise near the text elements. In order to further improve the accuracy of text and non-text separation, the following process can be used to find and separate non-text elements near the text region:

[0134] Determine whether the attribute characteristics of each connected area in the third connected area set meet the second non-text element condition; determine whether the adjacent position information of each connected area in the third connected area set meets the third non-text element condition; determine the connected areas in the third connected area set that do not meet the second non-text element condition, and the connected areas that meet the second non-text element condition but do not meet the third non-text element condition as connected areas of text elements.

[0135] The above process can further improve the accuracy of text-non-text separation.

[0136] The above-mentioned second non-text condition includes the first condition and the second condition; or, the first condition and the third condition; or, the first condition, the second condition and the third condition.

[0137] Among them, the first condition can be expressed as A i =max(Ω1)∩A i >k1×median(Ω1);

[0138] The second condition can be expressed as H i=max(Ω2)∩H i >k2×median(Ω2);

[0139] The third condition can be expressed as W i =max(Ω3)∩W i >k3×median(Ω3).

[0140] In the formula,

[0141] Ω1 represents the set of pixel numbers of each connected region in the third connected region set;

[0142] Ω2 represents the set of heights of each connected region in the third connected region set;

[0143] Ω3 represents the set of widths of each connected region in the third connected region set;

[0144] Mean means calculating the average value, and median means calculating the median;

[0145] A i Represents CC in the third connected region set i The number of pixels in (the i-th connected region);

[0146] H i Represents CC in the third connected region set i The height of the minimum bounding rectangle of ;

[0147] W i Represents CC in the third connected region set i The width of the minimum bounding rectangle of the .

[0148] The third non-text element condition mentioned above includes:

[0149] The distance between each connected area and the adjacent connected area corresponding to each connected area is greater than or equal to a preset distance; and / or

[0150] The second number of adjacent connected regions corresponding to each connected region is greater than or equal to a second value, and the second number is the maximum value between the number of left connected regions and the number of right connected regions.

[0151] The third non-text element condition mentioned above can be expressed by the following two conditional formulas:

[0152] Conditional formula 1: min(LNWS i ,RNWS i )≥10×max(medianWS,meanWS)

[0153] Conditional formula 2: max(numLN i, numRN i ) ≥ 3

[0154] Among them, conditional formula 1 can be understood as that if the distance between the connected regions in the fourth connected region set and their adjacent connected regions is too large, the connected regions are recognized as non-text element connected regions; conditional formula 2 can be understood as that if the height of the connected regions in the fourth connected region set is too large and spans three lines of text, the connected regions are regarded as non-text element connected regions.

[0155] As long as at least one of the following two conditional formulas is satisfied, the corresponding connected region can be recognized as a non-text element connected region.

[0156] Correspondingly, the connected regions with a distance less than the preset distance and a second quantity less than the second value are determined as text element connected regions.

[0157] Optionally, for Chinese documents, conditional formula 2 can be changed to: max(numLN i , numRN i ) > 4 to prevent misrecognition of Chinese documents. Figure 6 This is a schematic diagram of the blank area analysis of Chinese documents provided by the embodiments of the present invention. As Figure 6 shown, without considering Figure 5 (c), 5(d), Figure 6 in (a), the character "受" has 5 connected regions, the character "身" has 1 connected region, and the left connected regions of the 4 connected regions of the character "受" are all the character "身". According to max(numLN i , numRN i ) > 3, the character "身" will be misjudged as a non-text element connected region. When considering Figure 5 (c), 5(d), Figure 6 in (a), the connected regions inside the character "受" are left and right connected regions with each other, and only one connected region of the character "受" has the left connected region as the character "身", which does not satisfy max(numLN i , numRN i ) > 3. Therefore, the character "身" will not be misjudged as a non-text element connected region.

[0158] Without considering Figure 5 (b), (d), Figure 6 in (b), the character "求" has 4 connected regions, the character "要" has 1 connected region, and the left connected regions of the 4 connected regions of the character "求" are all the character "要". According to max(numLN i , numRN i ) > 3, the character "要" will be misjudged as a non-text element connected region. When considering Figure 5In the cases of (b) and 5(d), Figure 6 In (b), the connected regions inside the character "求" are left and right connected regions to each other. The left connected region of the only connected region of the character "求" is the character "要", which does not satisfy max(numLN i , numRN i ) > 3. Therefore, the character "要" will not be misjudged as a connected region of a non-text element.

[0159] In summary, blank region analysis can prevent Chinese characters from being misjudged as connected regions of non-text elements.

[0160] Optionally, in some embodiments, if the connected regions in the third set of connected regions simultaneously satisfy the above first condition and the second condition, or, simultaneously satisfy the above first condition and the third condition, or, simultaneously satisfy the above three conditions, then the connected region can be used as a candidate connected region of a non-text element. If the connected regions in the third set of connected regions do not satisfy any combination of the conditions in the second non-text element condition, then the connected region is determined as a connected region of a text element.

[0161] Furthermore, further analysis can be performed on the above candidate connected regions of non-text elements to determine whether the adjacent position relationship of the above candidate connected regions of non-text elements satisfies the third non-text element condition. If the third non-text element condition is not satisfied, then the connected region is determined as a connected region of a text element. If the third non-text element condition is satisfied, then the connected region is determined as a connected region of a non-text element.

[0162] Figure 7 It is a schematic diagram of the text-non-text separation result of the region separation method for document images provided by the embodiments of the present invention. As Figure 7 shown, Figure 7 (a) is the original document image, Figure 7 (b) is the connected region of the separated non-text elements, Figure 7 (c) is the separated text region. It can be seen that through the region separation method provided by this embodiment, more non-text elements (such as Figure 7 the bar-shaped non-text elements between the text regions in (a)) can be accurately extracted from the text region, improving the accuracy of text-non-text separation of document images.

[0163] The document image region separation method provided by the embodiment of the present invention obtains a binary image by preprocessing the document image, performs connected region analysis on the binary image to obtain a first connected region set, and divides the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set, wherein the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements and connected regions of non-text elements identified as text elements. Further, according to the attribute characteristics and adjacent position information of each connected region in the third connected region set, the connected regions of text elements are determined from the third connected region set. The above method can extract more non-text elements from the text region, thereby improving the accuracy of text-non-text separation of the document image.

[0164] Figure 8 A schematic flow chart of a method for separating regions of document images provided by another embodiment of the present invention. Figure 1 Based on the embodiment shown, Figure 8 As shown, after step 106, the following further includes:

[0165] Step 201: Obtain the first number of pixels of the minimum circumscribed rectangle of the connected area overlapping with the connected area of ​​the non-text element.

[0166] The connected regions of the non-text elements in this embodiment include: the connected regions of the non-text elements in the second connected region set determined in step 104 of the above embodiment, and the connected regions of the non-text elements determined in step 106 of the above embodiment.

[0167] The connected areas where the connected areas of the non-text elements overlap refer to the connected areas within a preset range near the connected areas of the non-text elements, including Figure 5 (a), 5(b), 5(c), and 5(d). It should be noted that Figure 5 If the distance between the two connected regions in (a) is less than or equal to the preset distance, it can be regarded that the two connected regions overlap.

[0168] After determining the connected area overlapping with the connected area of ​​the non-text element, the pixel value of the minimum circumscribed rectangle corresponding to the connected area is obtained, that is, the first pixel number.

[0169] Step 202: Obtain the second number of pixels after the connected area of ​​the non-text element is dilated.

[0170] In this embodiment, a smaller kernel is used to perform a dilation operation on the connected area of ​​non-text elements, where the kernel size is (mf, mf), mf is a value related to the image resolution, mf = min(h, w) / 200, and mf is usually in the range of [3, 10], where h and w represent the height and width of the document image, respectively.

[0171] After the expansion operation is performed on the connected area of ​​the non-text element, the pixel value of the minimum circumscribed rectangle corresponding to the expanded connected area of ​​the non-text element is obtained, that is, the second pixel number.

[0172] Step 203: If the first pixel number is greater than the second pixel number, the connected area overlapping with the connected area of ​​the non-text element is determined as the connected area of ​​the text element.

[0173] Correspondingly, if the first number of pixels is less than or equal to the second number of pixels, the connected area overlapping with the connected area of ​​the non-text element is determined as the connected area of ​​the non-text element.

[0174] The above process can be regarded as a noise elimination process, which can correctly determine the smaller connected areas near the connected areas of non-text elements as the connected areas of non-text elements, and other connected areas that do not meet the above conditions as the connected areas of text elements, thereby further improving the accuracy of text area detection in document images.

[0175] The document image region separation method provided in this embodiment further identifies and filters smaller noise regions near the non-text elements after identifying and filtering the non-text elements of the document image, thereby obtaining a more accurate text region.

[0176] Fig. 9 Schematic diagram of the functional structure of a document image region separation device provided by an embodiment of the present invention. Fig. 9 As shown, the document image region separation device 300 provided in this embodiment includes:

[0177] An image preprocessing module 301 is used to preprocess the document image to obtain a binary image;

[0178] A connected region analysis module 302, configured to perform a connected region analysis on the binary image to obtain a first connected region set;

[0179] An acquisition module 303 is used to acquire attribute features of each connected region in the first connected region set;

[0180] A connected region division module 304 is used to divide the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set; the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements;

[0181] The acquisition module 303 is further used to acquire attribute characteristics and adjacent position information of each connected area in the third connected area set;

[0182] The connected region division module 304 is further configured to determine the connected regions of the text elements from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set.

[0183] Optionally, the attribute features of each of the connected regions include the number of pixels, pixel density, aspect ratio of each of the connected regions, and the first number of other connected regions nested in the minimum circumscribed rectangle of each of the connected regions.

[0184] Optionally, the connected region division module 304 is specifically used to:

[0185] It is determined whether the attribute characteristics of each connected region in the first connected region set meet the first non-text element condition, and the connected regions whose attribute characteristics meet the first non-text element condition are divided into the second connected region set.

[0186] Optionally, the first non-text element condition includes at least one of the following conditions:

[0187] The number of pixels in the connected area is less than the preset number of pixels;

[0188] The pixel density of the connected area is less than the preset pixel density;

[0189] The aspect ratio of the connected area is smaller than the preset aspect ratio;

[0190] A first number of other connected regions nested in the minimum circumscribed rectangle of the connected region is greater than a first value.

[0191] Optionally, the connected region analysis module 302 is further used to:

[0192] Performing blank area analysis on each connected area in the third connected area set to determine adjacent connected areas corresponding to each connected area in the third connected area set;

[0193] The acquisition module 303 is specifically used to obtain adjacent position information corresponding to each connected area in the third connected area set, wherein the adjacent position information includes the distance between each connected area and the adjacent connected area corresponding to each connected area, the number of left connected areas and the number of right connected areas of each connected area.

[0194] Optionally, the connected region division module 304 is specifically used to:

[0195] Determining whether the attribute feature of each connected region in the third connected region set satisfies the second non-text element condition;

[0196] Determining whether adjacent position information of each connected region in the third connected region set satisfies a third non-text element condition;

[0197] The connected regions in the third connected region set that do not satisfy the second non-text element condition and the connected regions that satisfy the second non-text element condition but do not satisfy the third non-text element condition are determined as connected regions of text elements.

[0198] Optionally, the second non-text condition includes:

[0199] The first and second conditions; or,

[0200] The first condition and the third condition; or,

[0201] the first condition, the second condition and the third condition;

[0202] Among them, the first condition is A i =max(Ω1)∩A i >k1×median(Ω1); the second condition is H i =max(Ω2)∩H i >k2×median(Ω2); the third condition is W i =max(Ω3)∩W i >k3×median(Ω3);

[0203] In the formula, Ω1 represents the set of pixel numbers of each connected region in the third connected region set, Ω2 represents the set of heights of each connected region in the third connected region set, Ω3 represents the set of widths of each connected region in the third connected region set, mean represents the calculation of the mean, median represents the calculation of the median, A i represents the number of pixels of the i-th connected region in the third connected region set, H i represents the height of the minimum circumscribed rectangle of the i-th connected region in the third connected region set, Wi Represents the width of the minimum circumscribed rectangle of the ith connected region in the third connected region set.

[0204] Optionally, the third non-text element condition includes:

[0205] The distance between each connected area and the adjacent connected area corresponding to each connected area is greater than or equal to a preset distance; and / or

[0206] A second number of adjacent connected regions corresponding to each connected region is greater than or equal to a second value, and the second number is a maximum value between the number of left connected regions and the number of right connected regions.

[0207] Optionally, the acquisition module 303 is further used to acquire the first pixel number of the minimum circumscribed rectangle of the connected area overlapping with the connected area of ​​the non-text element;

[0208] The acquisition module 303 is further used to acquire the second pixel number after the connected area of ​​the non-text element is expanded;

[0209] The connected region division module 304 is further configured to: if the first number of pixels is greater than the second number of pixels, determine the connected region overlapping with the connected region of the non-text element as the connected region of the text element.

[0210] The document image region separation device provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0211] Fig.10 Schematic diagram of the hardware structure of the document image region separation device provided by one embodiment of the present invention. Fig.10 As shown, the document image region separation device 400 provided in this embodiment includes:

[0212] Memory 401;

[0213] Processor 402; and

[0214] Computer programs;

[0215] Among them, the computer program is stored in the memory 401 and is configured to be executed by the processor 402 to implement the technical solution of any of the aforementioned method embodiments. Its implementation principle and technical effect are similar and will not be repeated here.

[0216] Optionally, the memory 401 may be independent or integrated with the processor 402 .

[0217] When the memory 401 is a device independent of the processor 402, the document image region separation device 400 further includes:

[0218] The bus 403 is used to connect the memory 401 and the processor 402 .

[0219] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by the processor 402 to implement the various steps executed by the document image region separation device 400 in the above method embodiment.

[0220] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0221] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0222] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0223] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0224] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in the region separation device of the document image.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for region separation of a document image, characterized in that: include: Preprocess the document image to obtain a binary image; Using the skimage library in Python, a connected region analysis is performed on the binary image to obtain a first connected region set; Acquire and divide the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set; the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements and connected regions of non-text elements identified as text elements, wherein text elements include at least one of characters, numbers, symbols, and formulas in different languages, and each text element includes more than one connected region; Taking each connected region in the document image as a unit, obtaining and determining the connected regions of text elements from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set, so as to separate the connected regions of non-text elements in the third connected region set; The method further comprises: Acquire a first pixel number of a minimum circumscribed rectangle of a connected area overlapping with the connected area of ​​the non-text element; the connected area of ​​the non-text element includes the connected areas of the non-text elements in the second connected area set and the connected areas of the non-text elements in the third connected area set; Obtaining a second number of pixels after the connected area of ​​the non-text element is dilated; If the first number of pixels is greater than the second number of pixels, determining the connected area overlapping with the connected area of ​​the non-text element as the connected area of ​​the text element; Acquiring adjacent position information of each connected area in the third connected area set includes: Performing blank area analysis on each connected area in the third connected area set to determine adjacent connected areas corresponding to each connected area in the third connected area set; performing blank area analysis on each connected area in the third connected area set includes analyzing overlap and nesting of connected areas in the document image; Acquire adjacent position information corresponding to each connected area in the third connected area set, wherein the adjacent position information includes a distance between each connected area and an adjacent connected area corresponding to each connected area, and the number of left connected areas and the number of right connected areas of each connected area; Determining the connected regions of the text element from the third connected region set according to the attribute characteristics and adjacent position information of each connected region in the third connected region set includes: Determining whether the attribute feature of each connected region in the third connected region set meets the second non-text element condition; Determining whether adjacent position information of each connected region in the third connected region set satisfies a third non-text element condition; Determine the connected regions in the third connected region set that do not satisfy the second non-text element condition, and the connected regions that satisfy the second non-text element condition but do not satisfy the third non-text element condition as connected regions of text elements; The third non-text element condition includes: The distance between each connected area and the adjacent connected area corresponding to each connected area is greater than or equal to a preset distance; and / or The second number of adjacent connected areas corresponding to each connected area is greater than or equal to a second value, and the second number is the maximum value of the number of left connected areas and the number of right connected areas; the second number of adjacent connected areas corresponding to each connected area is greater than or equal to the second value, which is used to indicate that the height of the connected areas in the candidate non-text element connected area set is greater than a preset value, the candidate non-text element connected area set is a set of connected areas that meet the second non-text element condition, and the second value corresponding to the document image when it is a Chinese document image is greater than the second value corresponding to the document image when it is an English document image; The second non-text element condition includes: a first condition and a second condition; or, The first condition and the third condition; or, the first condition, the second condition and the third condition; Among them, the first condition is A i =max(Ω1)∩A i >k1×median(Ω1); The second condition is H i =max(Ω2)∩H i >k2×median(Ω2); The third condition is W i =max(Ω3)∩W i >k3×median(Ω3); In the formula, Ω1 represents the set of pixel numbers of each connected region in the third connected region set, Ω2 represents the set of heights of each connected region in the third connected region set, Ω3 represents the set of widths of each connected region in the third connected region set, Mean means calculating the average value. median means calculating the median, A i represents the number of pixels of the i-th connected region in the third connected region set, H i represents the height of the minimum circumscribed rectangle of the i-th connected region in the third connected region set, W i Represents the width of the minimum circumscribed rectangle of the ith connected region in the third connected region set.

2. The method according to claim 1, characterized in that The attribute features of each of the connected regions include the number of pixels, pixel density, aspect ratio of each of the connected regions, and the first number of other connected regions nested in the minimum circumscribed rectangle of each of the connected regions.

3. The method according to claim 1, characterized in that: The step of dividing the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set includes: It is determined whether the attribute characteristics of each connected region in the first connected region set meet the first non-text element condition, and the connected regions whose attribute characteristics meet the first non-text element condition are divided into the second connected region set.

4. The method according to claim 3, characterized in that The first non-text element condition includes at least one of the following conditions: The number of pixels in the connected area is less than the preset number of pixels; The pixel density of the connected area is less than the preset pixel density; The aspect ratio of the connected area is smaller than the preset aspect ratio; A first number of other connected regions nested in the minimum circumscribed rectangle of the connected region is greater than a first value.

5. A document image region separation device, characterized in that: include: An image preprocessing module is used to preprocess the document image to obtain a binary image; A connected region analysis module, used to perform connected region analysis on the binary image using the skimage library in Python to obtain a first connected region set; An acquisition module, used for acquiring attribute characteristics of each connected region in the first connected region set; A connected region division module, used for dividing the first connected region set into a second connected region set and a third connected region set according to the attribute characteristics of each connected region in the first connected region set; the second connected region set includes connected regions of non-text elements, and the third connected region set includes connected regions of text elements and connected regions of non-text elements identified as text elements, wherein text elements include at least one of characters, numbers, symbols, and formulas in different languages, and each text element includes more than one connected region; The acquisition module is further used to acquire the attribute characteristics and adjacent position information of each connected area in the third connected area set, taking each connected area in the document image as a unit; The connected region division module is further used to determine the connected regions of text elements from the third connected region set based on the attribute characteristics and adjacent position information of each connected region in the third connected region set, taking each connected region in the document image as a unit, so as to separate the connected regions of non-text elements in the third connected region set; The acquisition module is further used to acquire the first pixel number of the minimum circumscribed rectangle of the connected area overlapping with the connected area of ​​the non-text element; the connected area of ​​the non-text element includes the connected areas of the non-text elements in the second connected area set and the connected areas of the non-text elements in the third connected area set; The acquisition module is further used to acquire the second pixel quantity after the connected area of ​​the non-text element is dilated; The connected region division module is further used to: if the first pixel number is greater than the second pixel number, determine the connected region overlapping with the connected region of the non-text element as the connected region of the text element; The connected region analysis module is further used for: Performing blank area analysis on each connected area in the third connected area set to determine adjacent connected areas corresponding to each connected area in the third connected area set; performing blank area analysis on each connected area in the third connected area set includes analyzing overlap and nesting of connected areas in the document image; The acquisition module is specifically used to acquire adjacent position information corresponding to each connected area in the third connected area set, wherein the adjacent position information includes the distance between each connected area and the adjacent connected area corresponding to each connected area, the number of left connected areas and the number of right connected areas of each connected area; The connected region division module is specifically used for: Determining whether the attribute feature of each connected region in the third connected region set meets the second non-text element condition; Determining whether adjacent position information of each connected region in the third connected region set satisfies a third non-text element condition; Determine the connected regions in the third connected region set that do not satisfy the second non-text element condition, and the connected regions that satisfy the second non-text element condition but do not satisfy the third non-text element condition as connected regions of text elements; The third non-text element condition includes: The distance between each connected area and the adjacent connected area corresponding to each connected area is greater than or equal to a preset distance; and / or The second number of adjacent connected areas corresponding to each connected area is greater than or equal to a second value, and the second number is the maximum value of the number of left connected areas and the number of right connected areas; the second number of adjacent connected areas corresponding to each connected area is greater than or equal to the second value, which is used to indicate that the height of the connected areas in the candidate non-text element connected area set is greater than a preset value, the candidate non-text element connected area set is a set of connected areas that meet the second non-text element condition, and the second value corresponding to the document image when it is a Chinese document image is greater than the second value corresponding to the document image when it is an English document image; The second non-text element condition includes: a first condition and a second condition; or, The first condition and the third condition; or, the first condition, the second condition and the third condition; Among them, the first condition is A i =max(Ω1)∩A i >k1×median(Ω1); The second condition is H i =max(Ω2)∩H i >k2×median(Ω2); The third condition is W i =max(Ω3)∩W i >k3×median(Ω3); In the formula, Ω1 represents the set of pixel numbers of each connected region in the third connected region set, Ω2 represents the set of heights of each connected region in the third connected region set, Ω3 represents the set of widths of each connected region in the third connected region set, Mean means calculating the average value. median means calculating the median, A i represents the number of pixels of the i-th connected region in the third connected region set, H i represents the height of the minimum circumscribed rectangle of the i-th connected region in the third connected region set, W i Represents the width of the minimum circumscribed rectangle of the ith connected region in the third connected region set.

6. A device for separating regions of a document image, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.

Images