A document element labeling method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of document processing, and an embodiment of the present application provides a document element labeling method and device, electronic equipment and storage medium, the above-mentioned method comprises the following steps: converting a document to be labeled into a target image; performing morphological processing on the target image to obtain a preliminary feature region corresponding to each element of the document to be labeled in the target image; determining pixel points in each preliminary feature region as a same target region if the pixel values of the pixel points belong to a same connected region; obtaining element content in the target region, and labeling the target region based on the element content. The method can not only reduce the consumption of artificial resources in the process of document element labeling, but also ensure the accuracy of the determined target region by using image processing methods such as morphological processing, that is, the precision of document element labeling can be ensured.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, and in particular to a document element annotation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of deep learning technology, document processing techniques based on deep learning have also been widely applied. For example, document element classification models trained using deep learning can be used to classify various elements in a document, such as text elements, image elements, and table elements. When training deep learning models related to document processing, it is usually necessary to label the document elements, and then use these labeled document elements as the training set for training the deep learning models.

[0003] Currently, commonly used methods for annotating training sets of deep learning document elements include:

[0004] Annotation Method 1: Based on the use of annotation software, manually determine the areas of each element in the document, and then manually annotate the different elements;

[0005] Annotation Method 2: Annotation is performed by combining deep learning models and manual annotation. This involves using manually annotated document elements as a training set to train a document element annotation model, and then using the document element annotation model to annotate a large number of document elements.

[0006] However, the first annotation method described above involves manually determining the regions of each element in the document to annotate different elements. This not only incurs high labor costs but also results in low annotation accuracy due to errors in manually determining the regions. The second annotation method itself has high costs for training the document element annotation model, and to ensure the accuracy of the annotation, the document element annotation model must be highly accurate. A high-precision document element annotation model means that a large number of high-quality training sets are required, and the training sets are also obtained through manual annotation. Therefore, the second annotation method also has the same problems as the first annotation method. Summary of the Invention

[0007] The purpose of this invention is to provide a document element annotation method, apparatus, electronic device, and storage medium to reduce the manual cost of document element annotation without affecting the accuracy of the annotation.

[0008] In one aspect of this invention, a document element annotation method is provided, comprising:

[0009] Convert the document to be annotated into a target image;

[0010] Morphological processing is performed on the target image to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image;

[0011] Pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region are identified as the same target region;

[0012] Obtain the element content in the target area, and annotate the target area based on the element content.

[0013] Optionally, the step of performing morphological processing on the target image to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image includes:

[0014] Convert the target image into a grayscale image;

[0015] The grayscale image is binarized based on a preset filtering kernel to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image.

[0016] Optionally, the step of binarizing the grayscale image based on a preset filtering kernel to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image includes:

[0017] For each pixel in the grayscale image, the new pixel value is obtained by subtracting the original pixel value from 255, thus obtaining the target grayscale image.

[0018] Based on a preset line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain a target morphological image;

[0019] Based on a preset region detection filter kernel, the target morphological image is subjected to erosion and dilation processing to obtain the preliminary feature regions corresponding to each element of the document to be annotated.

[0020] Optionally, the step of performing erosion and dilation processing on the target grayscale image based on a preset line detection filter kernel to obtain a morphological image includes:

[0021] Based on a preset vertical line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain a preliminary morphological image;

[0022] The preliminary morphological image is subjected to erosion and dilation processing based on a preset horizontal line detection filter kernel to obtain the target morphological image.

[0023] Optionally, converting the document to be annotated into a target image includes:

[0024] The document to be annotated is converted into a target image using the document processing tool PyMuPDF.

[0025] Optionally, determining pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region as the same target region includes:

[0026] Pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 4-connected region are identified as the same target region, or pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 8-connected region are identified as the same target region.

[0027] Optionally, obtaining the element content in the target region and annotating the target region based on the element content includes:

[0028] Determine the location coordinates of the target area;

[0029] Obtain the element content at the position corresponding to the given coordinates in the target image;

[0030] Determine the type of the element content, and label the target region based on the type.

[0031] Optionally, determining the type of the element content and labeling the target region based on the type includes:

[0032] The type of the element content is determined to be document header, document footer, text paragraph, image, table, or formula, and the target area is labeled with the corresponding type of the element content.

[0033] In another aspect of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0034] Memory, used to store computer programs;

[0035] A processor, when executing a program stored in memory, implements the steps of any of the document element annotation methods described above.

[0036] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the document element annotation methods described above.

[0037] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the document element annotation methods described above.

[0038] The document element annotation method provided in this invention converts the document to be annotated into a target image; morphological processing is performed on the target image to obtain preliminary feature regions corresponding to each element of the document to be annotated; pixels whose pixel values ​​belong to the same connected region in each preliminary feature region are identified as the same target region; the element content in the target region is obtained, and the target region is annotated based on the element content. In other words, by performing morphological processing on the target image, the region division results that meet human visual standards in the target image can be obtained. Then, through connected component analysis, the target regions where different document elements are located can be determined. Finally, the target regions can be directly annotated based on the element content in the target regions. This not only reduces the manual resource consumption in the document element annotation process, but also ensures the accuracy of the determined target regions, i.e., the accuracy of document element annotation, due to the inherent superiority of traditional morphological image processing methods in region determination. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0040] Figure 1 A flowchart of a document element annotation method provided in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of a morphological processing method for a target image provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating morphological processing of an image according to an embodiment of the present invention;

[0043] Figure 4 This is another schematic diagram illustrating morphological processing of an image according to an embodiment of the present invention;

[0044] Figure 5 This is another schematic diagram illustrating morphological processing of an image provided by an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of a cross-shaped structural element;

[0046] Figure 7 This is a schematic diagram of the superposition of a cross-shaped structural element and an image pixel matrix.

[0047] Figure 8 A flowchart for document element annotation provided in an embodiment of the present invention;

[0048] Figure 9This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0050] Existing document annotation methods suffer from high labor costs and low accuracy due to errors in manually determining regions. To reduce the labor costs of document element annotation without affecting its accuracy, this invention provides a document element annotation method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0051] The document element annotation method provided in this embodiment of the invention will be described below. This method can be applied to any electronic device with image processing and document processing capabilities, and is not specifically limited thereto.

[0052] Figure 1 A flowchart of a document element annotation method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0053] Step 101: Convert the document to be annotated into a target image.

[0054] In this embodiment of the invention, the documents to be annotated include, but are not limited to: documents in ppt format, pptx format, txt format, doc format, docx format, xls format, xlsx format, and pdf format.

[0055] In this step, the document processing tool PyMuPDF can be used to convert the document to be annotated into the target image. Of course, any other tool capable of document-to-image conversion can also be used to convert the document to be annotated into the target image; no specific limitation is made here.

[0056] Step 102: Perform morphological processing on the target image to obtain the preliminary feature regions of each element of the document to be annotated in the target image.

[0057] In this embodiment of the invention, the elements of the document to be annotated may include: document header, document footer, text paragraphs, images, tables, and formulas, etc. For documents with annotations, the elements of the document to be annotated may also include annotation text.

[0058] The preliminary feature regions are the areas with image features in the target image obtained after morphological processing. Areas containing elements such as text paragraphs and table headers in the target image possess image features, so morphological processing can identify these areas. Large blank background areas, however, are not included in the preliminary feature regions.

[0059] Step 103: Pixels whose pixel values ​​in each preliminary feature region belong to the same connected region are identified as the same target region.

[0060] If the pixel values ​​of certain pixels belong to the same connected region, it means that these pixels are likely pixels corresponding to the same element. For example, for a piece of text, the pixel values ​​of its corresponding pixels are the same, belonging to the same connected region. Therefore, pixels whose pixel values ​​belong to the same connected region in each preliminary feature region can be identified as the same target region, and the pixels included in the same target region belong to the same element.

[0061] In this embodiment of the invention, pixels whose pixel values ​​in each preliminary feature region belong to the same 4-connected region can be determined as the same target region, or pixels whose pixel values ​​in each preliminary feature region belong to the same 8-connected region can be determined as the same target region.

[0062] For example, if the pixel values ​​of all pixels in the same 4-connected region are within the first preset pixel value range, then the pixels in the 4-connected region can be identified as the same target region. If the pixel values ​​of all pixels in the same 8-connected region are within the second preset pixel value range, then the pixels in the 8-connected region can be identified as the same target region. The first preset pixel value range and the second preset pixel value range can be set according to the actual application scenario. For example, the first preset pixel value range can be set to [245, 255] or [250, 255], and the second preset pixel value range can be set to [252, 255] or [253, 255].

[0063] Step 104: Obtain the element content in the target area and annotate the target area based on the element content.

[0064] In this embodiment of the invention, the element content refers to the content at the location corresponding to the position coordinates of the target area, and may include document headers, document footers, text paragraphs, images, tables, or formulas, etc. After obtaining the element content in the target area, the target area is marked based on the position coordinates of the element content in the target area to obtain the position information corresponding to the target area.

[0065] The target area can be a rectangular area. The labeling of the target area can specifically record the position coordinates, element content and element type of the rectangular area. The position coordinates can be the coordinates of the four corner points of the rectangular area, or the coordinates of one corner point of the rectangular area, as well as the height and width of the rectangular area. As long as the position of the rectangular area can be identified, there is no specific limitation here.

[0066] For example, if the target region 'a' contains a piece of text, then it can be labeled and the following information recorded: location coordinates: location coordinates of target region 'a'; element content: text content; element type: text.

[0067] The document element annotation method provided in this invention converts the document to be annotated into a target image; morphological processing is performed on the target image to obtain preliminary feature regions corresponding to each element of the document to be annotated; pixels whose pixel values ​​belong to the same connected region in each preliminary feature region are identified as the same target region; the element content in the target region is obtained, and the target region is annotated based on the element content. In other words, by performing morphological processing on the target image, the region division results that meet human visual standards in the target image can be obtained. Then, through connected component analysis, the target regions where different document elements are located can be determined. Finally, the target regions can be directly annotated based on the element content in the target regions. This not only reduces the manual resource consumption in the document element annotation process, but also ensures the accuracy of the determined target regions, i.e., the accuracy of document element annotation, due to the inherent superiority of traditional morphological image processing methods in region determination.

[0068] In one possible implementation, Figure 2 This is a flowchart of a morphological processing method for a target image provided in an embodiment of the present invention, such as... Figure 2 As shown, the morphological processing of the target image to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image may include:

[0069] Step 201: Convert the target image into a grayscale image.

[0070] Because grayscale images can display more details, improve image contrast, selectively highlight features of interest or suppress unwanted features, and make pixel distribution more uniform, target images can be converted into grayscale images for further processing.

[0071] Specifically, for each pixel in the target image, the pixel value can be transformed based on the formula Gray = R * 0.299 + G * 0.587 + B * 0.114, so that the transformed pixel value is located in [0, 255]. Here, R, G, and B represent the colors of the red, green, and blue channels, respectively, and Gray is the transformed pixel grayscale.

[0072] Step 202: Binarize the grayscale image based on the preset filter kernel to obtain the preliminary feature regions corresponding to each element of the document to be annotated in the target image.

[0073] In this embodiment, after binarizing the grayscale image based on a preset filter kernel, a processed image with only two pixel values, 0 and 255, is obtained. This results in more pronounced contrast, which is beneficial for identifying the preliminary feature regions corresponding to each element. In one case, if the document to be annotated includes column lines, the preliminary feature regions corresponding to each element can be the areas divided by the column lines. In another case, if the document to be annotated does not include column lines, the preliminary feature regions corresponding to each element can be regions with image features such as document headers, document footers, text paragraphs, images, tables, and formulas.

[0074] Specifically, the step of binarizing the grayscale image based on a preset filtering kernel to obtain the preliminary feature regions corresponding to each element of the document to be labeled in the target image may include the following steps A1-A3:

[0075] Step A1: For each pixel in the grayscale image, the new pixel value is obtained by subtracting the original pixel value from 255, thus obtaining the target grayscale image.

[0076] In a typical grayscale image, each element's corresponding pixel is black, and the background is white. Since white has higher brightness, facilitating recognition and other processing, a grayscale image can be inverted. This is done by subtracting the original pixel value from 255, and using the resulting value as the new pixel value to obtain the target grayscale image. In the target grayscale image, each element's corresponding pixel is white, and the background is black.

[0077] Step A2: Based on the preset line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain the target morphological image.

[0078] Step A3: Based on the preset region detection filter kernel, the target morphological image is subjected to erosion and dilation processing to obtain the preliminary feature regions corresponding to each element of the document to be annotated.

[0079] Specifically, erosion can be achieved by taking the minimum grayscale value within the neighborhood of each pixel as the output grayscale value for that pixel. Dilation, on the other hand, takes the maximum grayscale value within the neighborhood of each pixel as the output grayscale value for that pixel. In steps A2 and A3 above, different sizes of filter kernels and the number of processing iterations can be selected for erosion and dilation based on actual needs. For clarity and to illustrate the solution, examples will be provided later.

[0080] In one embodiment, the step of performing erosion and dilation processing on the target grayscale image based on a preset line detection filter kernel to obtain a morphological image may include the following steps B1-B2:

[0081] Step B1: Based on the preset vertical line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain a preliminary morphological image.

[0082] In this embodiment of the invention, the preset vertical line detection filter kernel can be used to detect whether there are column vertical lines in the document to be annotated represented by the target grayscale image. Specifically, it can be used for document layout analysis represented by the target image. The preset vertical line detection filter kernel can be set to a rectangular filter kernel of size m×1 according to the actual application, where m is the height of the rectangular filter kernel, 1 is the width of the rectangular filter kernel, and the value of m can be any reasonable value such as 2, 3, 4, etc., without limitation here. Specifically, the target grayscale image can be first eroded using the m×1 rectangular filter kernel to remove noise from the target grayscale image, and then the image obtained after erosion can be dilated using the m×1 rectangular filter kernel to form multiple closed regions, thereby obtaining a preliminary morphological image with multiple closed regions.

[0083] Erosion processing takes the minimum grayscale value within the neighborhood of each pixel as its output grayscale value. In other words, when eroding a target grayscale image using an m×1 rectangular filter kernel, for each pixel in the target grayscale image, the minimum grayscale value within its m×1 neighborhood is used as the new grayscale value for that pixel. Therefore, if any pixel within the current m×1 rectangle contains a black pixel, then the new grayscale value of that pixel will be 0.

[0084] Since the width of the rectangular filter kernel is 1, which is in units of pixels, when there are column lines in the target grayscale image, for the pixels on the column lines, all the pixels included in the m×1 rectangle are white pixels. Therefore, after the erosion process, the pixel values ​​of the pixels included in the column lines are still white, thus detecting the column lines in the target grayscale image.

[0085] Dilation is a process that takes the maximum value within the neighborhood of a pixel at a given location as the output grayscale value for that pixel. In other words, when dilating a target grayscale image using an m×1 rectangular filter kernel, for each pixel in the target grayscale image, the maximum grayscale value within its m×1 neighborhood is used as the new grayscale value for that pixel. Therefore, if any pixel within the current m×1 rectangle contains a white pixel, then the new grayscale value of that pixel will be 255.

[0086] Since the width of the rectangular filter kernel is 1, which is in units of pixels, when there are column lines in the target grayscale image, due to factors such as image noise, in some cases, the pixels included in the m×1 rectangle may be black. After dilation processing, the pixel values ​​of the pixels included in the column lines are all white, thus making it possible to detect the column lines in the target grayscale image more accurately.

[0087] For example, Figure 3 This is a schematic diagram illustrating morphological processing of an image according to an embodiment of the present invention, such as... Figure 3 As shown, the black lines in the target grayscale image 301 represent the column vertical lines in the document corresponding to the target grayscale image 301. By using a preset vertical line detection filter kernel to detect the target grayscale image 301, the lines shown in image 302 can be detected. These lines are the column vertical lines in the target grayscale image 301.

[0088] Step B2: Based on the preset horizontal line detection filter kernel, the preliminary morphological image is subjected to erosion and dilation processing to obtain the target morphological image.

[0089] In this embodiment of the invention, the preset horizontal line detection filter kernel can be used to detect whether there are column lines or open tables in the document to be annotated represented by the target image. The preset horizontal line detection filter kernel can be set to a rectangular filter kernel of size 1×n according to the actual application, where 1 is the height of the rectangular filter kernel, n is the width of the rectangular filter kernel, and the value of n can be any reasonable value such as 1, 2, 3, etc., without limitation here. Specifically, the preliminary morphological image can be eroded first using the rectangular filter kernel of size 1×n to remove noise from the preliminary morphological image, and then the image obtained after erosion can be dilated using the rectangular filter kernel of size 1×n to further strengthen the closed regions in the image, thereby obtaining the target morphological image.

[0090] Erosion processing takes the minimum gray value within the neighborhood of each pixel as its output gray value. In other words, when eroding the preliminary morphological image using a 1×n rectangular filter kernel, for each pixel in the preliminary morphological image, the minimum gray value within its 1×n neighborhood is used as the new gray value for that pixel. Therefore, if any pixel within the 1×n rectangle contains a black pixel, its new gray value will be 0.

[0091] Since the height of the rectangular filter kernel is 1, which is in units of pixels, when there are column lines in the preliminary morphological image, for the pixels on the column lines, the pixels included by the 1×n rectangle are all white pixels. Therefore, after erosion, the pixel values ​​of the pixels included by the column lines are still white, thus detecting the column lines in the preliminary morphological image.

[0092] Dilation is achieved by taking the maximum value within the neighborhood of each pixel as the output grayscale value for that pixel. In other words, when dilating a preliminary morphological image using a 1×n rectangular filter kernel, the maximum grayscale value within its 1×n neighborhood is used as the new grayscale value for each pixel. Therefore, if any white pixel exists within the 1×n rectangle, its new grayscale value will be 255.

[0093] Since the height of the rectangular filter kernel is 1, which is in units of pixels, when there are column lines in the preliminary morphological image, due to factors such as image noise, in some cases, the pixels included in the 1×n rectangle may be black. After dilation processing, the pixel values ​​of the pixels included in the column lines are all white, thus enabling more accurate detection of column lines in the target grayscale image.

[0094] For example, Figure 4 This is another schematic diagram illustrating morphological processing of an image according to an embodiment of the present invention, such as... Figure 4 As shown, the preliminary morphological image 401 includes the document text content, column lines, open tables, and images. By performing erosion and dilation processing on the preliminary morphological image 401 using a preset line detection filter, the open table 402 and column lines 403 shown in the black rectangle can be obtained.

[0095] In this embodiment of the invention, the target grayscale image is subjected to erosion and dilation processing by a preset vertical line detection filter kernel and a preset horizontal line detection filter kernel. This process can obtain the column vertical lines and column horizontal lines in the document to be annotated represented by the target image. By adding the detected column vertical lines and column horizontal lines together, the closed table structure in the document to be annotated represented by the target image can be obtained.

[0096] When the target image represents a document with line structures, using a preset line detection filter kernel to perform line detection on the target image can greatly assist in document layout analysis.

[0097] When the target image represents a document to be annotated that does not contain line structures, but only consists of ordinary images, tables, and text regions such as headings and paragraphs, this embodiment of the invention can use a preset region detection filter kernel to perform erosion and dilation processing on the target morphological image to obtain preliminary feature regions corresponding to each element of the document to be annotated. The preset region detection filter kernel can be an m×n rectangular filter kernel selected according to the actual application scenario, and the values ​​of m and n can be adjusted according to the page size of the document to be annotated. That is, an m×n rectangular filter kernel can be used to first erode the target morphological image to further remove noise, and then the m×n rectangular filter kernel is used again to dilate the resulting image to further enhance closed regions in the image, thus obtaining preliminary feature regions corresponding to each element of the document to be annotated.

[0098] Figure 5 This is another schematic diagram illustrating morphological processing of an image provided by an embodiment of the present invention, such as... Figure 5 As shown, the target morphological image 501 is subjected to erosion and dilation processing using a preset region detection filter kernel. The resulting preliminary feature regions include multiple image regions 502, header region 503, and footer region 504, etc.

[0099] Morphological processing of images can be used to simplify pages, extract key feature regions, and obtain different areas of the page that conform to human visual standards. In this embodiment of the invention, appropriate preset horizontal line detection filter kernels, preset vertical line detection filter kernels, and preset region detection filter kernels can be selected based on the different document element features such as tables, images, and text paragraphs in the document to be annotated to perform erosion and dilation processing on the grayscale image, thereby obtaining masks corresponding to different document element features in the grayscale image, which serve as preliminary feature regions.

[0100] In this embodiment of the invention, the morphological erosion operation on the image can specifically be achieved by taking the minimum gray value within the neighborhood of each pixel as the output gray value of that pixel. The neighborhood structure of a pixel can be a rectangular structure, an elliptical structure, a cross-shaped structure, etc., and can be defined as a structuring element, which can actually be a 0-1 binary matrix.

[0101] For example, Figure 6 This is a schematic diagram of a cross-shaped structural element. Assume the pixel matrix of the target image is: Can Figure 6 The cross-shaped structural element shown is superimposed on this matrix to obtain the following: Figure 7 The superimposed schematic diagram shown below, in which, Figure 7 This is a schematic diagram of the superposition of a cross-shaped structural element and an image pixel matrix. Figure 7 The cross structure formed by the shaded areas is the cross-shaped structural element. Figure 6 The point (0,2) of the middle cross structure element corresponds to Figure 7 The grayscale value "11" in the target image shown corresponds to point (1,1). Figure 7 The grayscale value "234" in the target image shown corresponds to point (1,2). Figure 7 The grayscale value "21" in the target image shown corresponds to point (1,3). Figure 7 The grayscale value "67" in the target image shown corresponds to point (2,2). Figure 7 The grayscale value in the target image shown is "31". Figure 6 Processing the image grayscale corresponding to each point of the cross-shaped structural element means processing the image grayscale corresponding to that point in the target image. Therefore, when processing... Figure 6 When processing the image grayscale corresponding to point (1,2) of the cross-shaped structural element, that is, when... Figure 7 When processing the pixel containing the grayscale value "21" in the target image shown, the minimum value can be found within the cross-shaped neighborhood and assigned to the point (1,2). According to... Figure 7 It can be seen that the minimum value within the cross-shaped neighborhood of the grayscale value "21" in the target image is 11. Therefore, the value 11 can be assigned to the pixel containing the grayscale value "21" to obtain the new pixel matrix corresponding to the target image.

[0102] Based on the obtained new pixel matrix, it can be seen that the erosion operation reduces the grayscale value of the target image. That is, the overall brightness of the output image after erosion is lower than the brightness of the original target image. The area of ​​the brighter regions in the original target image will become smaller, and the area of ​​the darker regions will increase.

[0103] Morphological dilation of an image can be considered the inverse of erosion. Specifically, it involves taking the maximum value within the neighborhood of each pixel and using that value as the output grayscale value. Therefore, in the dilated image compared to the original image, brighter objects will appear larger, and darker objects will appear smaller. Similarly,... Figure 7 For example, in the context of Figure 6 When processing the image grayscale corresponding to point (1,2) of the cross-shaped structural element, that is, when... Figure 7 When processing the pixel containing the grayscale value "21" in the target image shown, the maximum value can be found within the cross-shaped neighborhood and assigned to point (1,2). For example... Figure 7 As shown, the minimum value within the cross-shaped neighborhood of the grayscale value "21" in the target image is 234. Therefore, 234 can be assigned to the pixel containing the grayscale value "21" to obtain the new pixel matrix corresponding to the target image. Based on the obtained new pixel matrix, it can be seen that the dilation operation increases the gray value of the target image. That is, the overall brightness of the output image after dilation is higher than the brightness of the original target image. The darker areas in the original target image will become smaller, and the brighter areas will become larger.

[0104] In this embodiment of the invention, the target image can be first eroded, and then the eroded image can be dilated to obtain the target morphological image. Alternatively, the target image can be first dilated, and then the dilated image can be eroded to obtain the target morphological image.

[0105] In one possible implementation, the step of determining pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region as the same target region may include: determining pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 4-connected region as the same target region, or determining pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 8-connected region as the same target region.

[0106] In other words, in the embodiments of the present invention, the grayscale image of the target image can be divided into page segments in the form of a mask after morphological erosion and dilation processing. Each preliminary feature region can be used as the foreground of the image, which is visible to the human eye as a pixel value of 255, to distinguish it from the background region corresponding to a pixel value of 0.

[0107] Specifically, the foreground pixels in the preliminary feature region image that are 4-connected or 8-connected can be divided into the same connected region, and the rectangular outer contour of the same connected region can be marked to obtain one or more closed connected regions. All one or more closed connected regions obtained can be used as the target region.

[0108] In one possible implementation, Figure 8 A flowchart for document element annotation provided in an embodiment of the present invention, such as Figure 8 As shown, obtaining the element content in the target region and annotating the target region based on the element content may include:

[0109] Step 801: Determine the location coordinates of the target area.

[0110] Specifically, after dividing the 4-connected or 8-connected foreground pixels in the preliminary feature region image into the same connected domain, and obtaining one or more closed connected domains as the target region, the rectangular outer contour of the target region can be marked, and the coordinate information of the rectangular outer contour can be obtained.

[0111] Step 802: Obtain the content of the element at the position corresponding to the position coordinate in the target image.

[0112] In this step, the content at that coordinate in the target image can be determined based on the coordinate information of the rectangular outer contour of the target area. Then, OCR (Optical Character Recognition) technology can be used to extract the element content at that location in the target image.

[0113] Alternatively, in this step, the element content at the corresponding position can be directly retrieved from the document to be annotated based on the coordinate information of the rectangular outer contour of the target area.

[0114] Step 803: Determine the type of element content and label the target area based on the type.

[0115] Specifically, you can determine the type of an element's content as a document header, document footer, text paragraph, image, table, or formula, and then label the target area with the corresponding type of the element's content. For example, if the element's content type is a document header, then the target area will be labeled as a document header; if the element's content type is a text paragraph, then the target area will be labeled as a text paragraph.

[0116] The method provided in this invention can utilize mathematical methods such as morphological detection and connected component analysis to achieve the regional division of various elements in the document to be annotated. The task of annotating document elements can be accomplished with a small amount of CPU computing resources. This not only reduces the consumption of manual resources in the process of annotating document elements, but also ensures the accuracy of the determined target region and the precision of the document element annotation due to the superiority of traditional image processing methods such as morphological processing in determining the region.

[0117] This invention also provides an electronic device, such as... Figure 9As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0118] Memory 903 is used to store computer programs;

[0119] When processor 901 executes a program stored in memory 903, it performs the following steps:

[0120] Convert the document to be annotated into a target image;

[0121] Morphological processing is performed on the target image to obtain preliminary feature regions corresponding to each element of the document to be labeled in the target image;

[0122] Pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region are identified as the same target region;

[0123] Obtain the element content in the target area, and annotate the target area based on the element content.

[0124] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0125] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0126] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0127] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0128] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the document element annotation methods described in the above embodiments.

[0129] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the document element annotation methods described in the above embodiments.

[0130] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0132] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0133] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A document element annotation method, characterized in that, include: Convert the document to be annotated into a target image; Convert the target image into a grayscale image; For each pixel in the grayscale image, the new pixel value is obtained by subtracting the original pixel value from 255, thus obtaining the target grayscale image. Based on a preset line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain a target morphological image; Based on a preset region detection filter kernel, the target morphological image is subjected to erosion and dilation processing to obtain the preliminary feature regions corresponding to each element of the document to be annotated; The preset line detection filter kernel is an m×1 or 1×n rectangular filter kernel, and the preset region detection filter kernel is an m×n rectangular filter kernel; Pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region are identified as the same target region; Obtain the element content in the target area, and annotate the target area based on the element content.

2. The method according to claim 1, characterized in that, The target grayscale image is subjected to erosion and dilation processing based on a preset line detection filter kernel to obtain a morphological image, including: Based on a preset vertical line detection filter kernel, the target grayscale image is subjected to erosion and dilation processing to obtain a preliminary morphological image; The preliminary morphological image is subjected to erosion and dilation processing based on a preset horizontal line detection filter kernel to obtain the target morphological image.

3. The method according to any one of claims 1-2, characterized in that, The process of converting the document to be annotated into a target image includes: The document to be annotated is converted into a target image using the document processing tool PyMuPDF.

4. The method according to any one of claims 1-2, characterized in that, The step of determining pixels whose pixel values ​​in each of the preliminary feature regions belong to the same connected region as the same target region includes: Pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 4-connected region are identified as the same target region, or pixels whose pixel values ​​in each of the preliminary feature regions belong to the same 8-connected region are identified as the same target region.

5. The method according to any one of claims 1-2, characterized in that, The step of obtaining the element content in the target region and annotating the target region based on the element content includes: Determine the location coordinates of the target area; Obtain the element content at the position corresponding to the given coordinates in the target image; Determine the type of the element content, and label the target region based on the type.

6. The method according to claim 5, characterized in that, The step of determining the type of the element content and labeling the target region based on the type includes: The type of the element content is determined to be document header, document footer, text paragraph, image, table, or formula, and the target area is labeled with the corresponding type of the element content.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-6.

Images