Sample processing method, apparatus, computer device, and storage medium

Abstract

The application relates to a sample processing method, device, computer equipment, storage medium and computer program product. The method comprises the following steps: inputting a plurality of sample images into a pre-trained table matching model to obtain a first table of the sample images; based on a plurality of standard tables and the first table, a plurality of second tables corresponding to the sample images are generated; wherein the second table of the sample images and the first table of the sample images constitute a plurality of standard tables corresponding to the sample images; and the first table and the second table are added to a sample data set. The method can generate a plurality of second tables based on the first table, can expand the sample data set, and is beneficial to improving the recognition accuracy of the table matching model.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a sample processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Real-world scenarios contain a large number of document images, including tables. The structured organization of tables facilitates information understanding and extraction. However, document images, including tables, are often stored in unstructured formats such as images. Applying table recognition technology can reduce table processing time; therefore, table recognition is an important research topic in the field of document understanding.

[0003] In traditional techniques, a table recognition model is typically obtained by training a large amount of table image data.

[0004] However, the lack of open-source table image datasets makes it difficult to support model training for table recognition. Summary of the Invention

[0005] Therefore, it is necessary to provide a sample processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can supplement the table image data in the table image dataset and support the optimization of the table recognition model, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a sample processing method. The method includes:

[0007] Multiple sample images are input into a pre-trained table matching model to obtain the first table of the sample images;

[0008] Based on multiple pre-set standard tables and a first table, multiple second tables corresponding to the sample images are generated; wherein, the second tables of the sample images and the first tables of the sample images constitute multiple standard tables corresponding to the sample images;

[0009] Add the first and second tables to the sample dataset.

[0010] In one embodiment, multiple sample images are input into a pre-trained table matching model to obtain a first table of the sample images, including:

[0011] Multiple sample images are input into a pre-trained table matching model to obtain the table layout information and the target field to be identified in the sample images; the target field belongs to at least one field in the layout information.

[0012] Based on the table layout information and the target fields to be identified, a first table of sample images is generated.

[0013] In one embodiment, multiple sample images are input into a pre-trained table matching model to obtain the layout information of the tables in the sample images, including:

[0014] Multiple sample images are input into a pre-trained table matching model to obtain the table layout in the sample images;

[0015] Perform field checks on the layout to determine the names and areas of each field in the layout.

[0016] Obtain the layout coordinate system corresponding to the layout. Based on the field name and field area, obtain the coordinate position of each field name in the layout coordinate system. Use each field name and its corresponding coordinate position in the layout coordinate system as layout information.

[0017] In one embodiment, based on a plurality of pre-set standard tables and a first table, a plurality of second tables corresponding to the sample image are generated, including:

[0018] Obtain the layout information of the table in the sample image;

[0019] Based on the layout information and multiple standard tables, multiple second tables corresponding to the sample images are generated.

[0020] In one embodiment, the layout information includes multiple table line segments and the positions of the table line segments; based on the layout information and multiple standard tables, multiple second tables corresponding to the sample image are generated, including:

[0021] For each standard table, obtain multiple standard line segments and the positions of the standard line segments of the standard table;

[0022] Retrieve multiple table line segments and their position coordinates from the layout information;

[0023] The system performs position matching between multiple table line segment positions and multiple standard line segment positions. If no standard line segment position matches the table line segment position, the position matching fails.

[0024] Erase the table segments corresponding to the positions of the table segments that failed to match from the first table to obtain the second table of the corresponding sample image.

[0025] In one embodiment, erasing the table line segment corresponding to the position of the table line segment where the position matching fails includes:

[0026] Get the position of the table line segment to be erased, and use it as the target object;

[0027] Get the colors of multiple pixels within a preset pixel range corresponding to the target object, and use them as the intermediate color;

[0028] Calculate the number of multiple intermediate colors and select the intermediate color with the largest number as the target color.

[0029] Erase the table lines based on the target color.

[0030] Secondly, this application also provides a sample processing apparatus. The apparatus includes:

[0031] The first table acquisition module is used to input multiple sample images into a pre-trained table matching model to obtain the first table of the sample images;

[0032] The second table generation module is used to generate multiple second tables corresponding to the sample image based on multiple pre-set standard tables and the first table; wherein, the second table of the sample image and the first table of the sample image constitute multiple standard tables corresponding to the sample image;

[0033] The sample dataset expansion module is used to add the first table and the second table to the sample dataset.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any of the above embodiments.

[0035] Fourthly, this application also provides a computer device readable storage medium. The computer device readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0037] The aforementioned sample processing method, apparatus, computer equipment, storage medium, and computer program product first input multiple sample images into a pre-trained table matching model to obtain a first table of the sample images. Then, based on multiple pre-set standard tables and the first table, multiple second tables corresponding to the sample images are generated; wherein, the second tables of the sample images and the first tables of the sample images constitute multiple standard tables for the corresponding sample images. Finally, the first and second tables are added to the sample dataset. By performing table matching on existing sample images to obtain the first table, and then generating multiple second tables based on the first table, the sample dataset can be expanded, which is beneficial to improving the recognition accuracy of the table matching model. Attached Figure Description

[0038] Figure 1 This is a diagram illustrating the application environment of a sample processing method in one embodiment.

[0039] Figure 2 This is a flowchart illustrating a sample processing method in one embodiment;

[0040] Figure 3 This is a schematic diagram of the first table in one embodiment;

[0041] Figure 4 This is a schematic diagram of a fully wireless table generated based on a first table in one embodiment;

[0042] Figure 5 This is a schematic diagram of a table with missing vertical lines on the outer border, generated based on a first table in one embodiment.

[0043] Figure 6 This is a schematic diagram of a three-line table generated based on a first table in one embodiment;

[0044] Figure 7 This is a schematic diagram of a table with missing all horizontal lines generated based on a first table in one embodiment;

[0045] Figure 8 This is a structural block diagram of a sample processing device in one embodiment;

[0046] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The sample processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, client 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on another network server. Server 104 provides a sample processing environment for client 102. First, server 104 can acquire multiple sample images from client 102 and input these images into a pre-trained table matching model to obtain a first table of the sample images. Then, based on multiple pre-set standard tables and the first table, server 104 generates multiple second tables corresponding to the sample images; wherein the second tables of the sample images and the first tables of the sample images constitute multiple standard tables for the corresponding sample images. Finally, server 104 adds the first and second tables to client 102's sample dataset.

[0049] The server 104 obtains multiple sample images from the client 102. These images can be sample images from the client 102's database (such as a sample dataset) or data generated on a daily basis on the client (i.e., business images generated in the production environment). The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0050] The sample processing method provided in this application can be applied to a single-side implementation on the server or client, or it can be applied to a system including both a client and a server, implemented through the interaction between the client and the server.

[0051] In one embodiment, such as Figure 2 As shown, a sample processing method is provided. Taking the application of this method to the server side as an example, the method includes the following steps 202 to 206.

[0052] Step 202: Input multiple sample images into the pre-trained table matching model to obtain the first table of the sample images.

[0053] In this embodiment, the server can input multiple sample images into a pre-trained table matching model to locate multiple first tables in each sample image. The first table is a fully wired table.

[0054] In this embodiment, as Figure 3 As shown, the server locates multiple first tables in each sample image using a pre-trained table matching model. This can include: inputting multiple sample images into the pre-trained table matching model to locate the position of each first table in the sample image; cropping the first table based on its position in the sample image; and generating a table image corresponding to the first table.

[0055] In this embodiment, the method may further include generating an identifier for the table image corresponding to the first table. The identifier may include, but is not limited to, label names, label coordinates, etc. One table image may correspond to multiple identifiers, and each identifier may correspond to a cell in the first table. For example, an identifier may correspond to the coordinates of the outer border of a cell in the first table, as well as the text information within that cell.

[0056] In another embodiment, the server can use pymupdf, a Python-based PDF (Portable Document Format) reader, to obtain page information for each page of the PDF. This page information may include, but is not limited to: text and its location within the PDF, images and their locations within the PDF, line segments and their locations within the PDF, etc.

[0057] In this embodiment, after the server obtains the page information of each page of the PDF through pymupdf, it can convert each page of the PDF into SVG (Scalable Vector Graphics) using pymupdf's built-in `get_svg_image` function. The server can iterate through the SVG corresponding to each page of the PDF and parse the SVG for each page. Furthermore, the server can construct an SVG soup object, that is, use Beautiful Soup to convert a complex HTML (HyperText Markup Language) document into a complex tree structure.

[0058] Step 204: Based on the pre-set multiple standard tables and the first table, generate multiple second tables corresponding to the sample image; wherein, the second tables of the sample image and the first table of the sample image constitute multiple standard tables corresponding to the sample image.

[0059] In this embodiment, the first table can be a fully wired table. The server can erase the table lines in the first table to obtain multiple second tables whose table lines are different from those of the first table.

[0060] In this embodiment, as Figures 4-7 As shown, the table type of the second table may include, but is not limited to: a table with no border, a table with no outer border, a table with no vertical line in the outer border, a three-line table, a table with no horizontal lines, a table with random defects, etc.

[0061] In another embodiment, the table type of the first table may not be a fully lined table. The server can input multiple sample images into a pre-trained table matching model to obtain a first table of the sample images and the table type of the first table. Furthermore, based on the table type of the first table, the server can erase or supplement the table lines in the first table to obtain multiple second tables whose table lines or table types are different from the first table.

[0062] Step 206: Add the first table and the second table to the sample dataset.

[0063] In this embodiment, the server can add the first table and multiple second tables corresponding to the PDF to the sample dataset, which can expand the training set of the above table matching model and be used to iteratively train the above table matching model to optimize the table matching model.

[0064] In the above sample processing method, firstly, multiple sample images are input into a pre-trained table matching model to obtain a first table for the sample images. Then, based on multiple pre-set standard tables and the first table, multiple second tables corresponding to the sample images are generated; wherein, the second tables of the sample images and the first tables of the sample images constitute multiple standard tables for the corresponding sample images. Finally, the first and second tables are added to the sample dataset. By performing table matching on existing sample images to obtain the first table, and then generating multiple second tables based on the first table, the sample dataset can be expanded, which is beneficial to improving the recognition accuracy of the table matching model.

[0065] In some embodiments, inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images may include: inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images and the target field to be identified; the target field belongs to at least one field in the layout information; and generating the first table of the sample images based on the layout information of the table and the target field to be identified.

[0066] In this embodiment, the first table is generated using a pre-trained table matching model. The training method for the table matching model includes: acquiring a training set containing multiple training samples, each corresponding to a table image; identifying the table images in each training sample using the initial table matching model to generate the first table corresponding to the training sample; determining the model loss of the initial table matching model based on the output first table and the training set; and iteratively training the initial table matching model according to the model loss to obtain the trained table matching model. The training set may include multiple samples from a sample dataset and data generated daily by the client (i.e., business images generated in the production environment). It is understood that business images generated offline in the production environment include, but are not limited to, images corresponding to text information and table images.

[0067] In some embodiments, inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images may include: inputting multiple sample images into a pre-trained table matching model to obtain the layout of the table in the sample images; performing field detection on the layout to determine the name of each field and the area of ​​each field in the layout; obtaining the layout coordinate system corresponding to the layout, and obtaining the coordinate position of each field name in the layout coordinate system based on each field name and each field area, and using each field name and the coordinate position of each field name in the layout coordinate system as the layout information.

[0068] In this embodiment, after parsing the SVG corresponding to each page and constructing the SVG's soup object, the server can also obtain the size of each group of text on that page and its coordinates (coordinates of four points: top, bottom, left, and right) on the current PDF page. Further, based on the size of each group of text on the page and its coordinates on the current PDF page, the server can delete text, images, and other information from the soup object, retaining only the line segments. Then, the server can convert the SVG text into a binary image and use the `cv2.findContours()` function to obtain the contour pixel coordinates of each matrix in the binary image. Where the height of each matrix frame is less than the size of the smallest text in the SVG, that matrix frame cannot be a table cell (because it cannot hold text) and can be deleted. Matrix frames containing other matrix frames are also deleted, retaining only a single, independent matrix frame.

[0069] In another embodiment, the server can group all the aforementioned matrix frames based on their positions on the PDF, grouping those belonging to the same first table. The server can iterate through the matrices, determining whether any two matrices are connected; if connected, they belong to the same first table.

[0070] In another embodiment, the server can delete the first table with 1 matrix frame (a first table can have two or more matrix frames) according to the grouping of the matrix. The remaining first tables can obtain the coordinates of the four outward vertices of the outermost matrix frame of each first table, and a screenshot can be taken to generate a table image corresponding to the first table.

[0071] In some embodiments, generating multiple second tables corresponding to a sample image based on a pre-set set of multiple standard tables and a first table may include: obtaining layout information of the tables in the sample image; and generating multiple second tables corresponding to the sample image based on the layout information and multiple standard tables.

[0072] In this embodiment, the layout information may include, but is not limited to, the identifier of the table image corresponding to the first table.

[0073] In this embodiment, the identifiers for the table images corresponding to the first table may include, but are not limited to, label names and label coordinates. The label names may correspond to the names of each field in the first table, and the label coordinates may correspond to the coordinate positions of each field name in the layout coordinate system.

[0074] In another embodiment, a label coordinate can correspond to the coordinate position of the outer frame of a cell in the first table in the layout coordinate system. For example, the representation of the identifier corresponding to the table image of the first table can be: [{"tokens":"A","bbox":[x1, y1, x2, y2]}], where bbox represents the cell and tokens represent the text content within the cell. Further, in "bbox":[x1, y1, x2, y2], x1 represents the x-coordinate of the top-left corner of the cell's outer frame, y1 represents the y-coordinate of the top-left corner of the cell's outer frame, x2 represents the x-coordinate of the bottom-right corner of the cell's outer frame, and y2 represents the y-coordinate of the bottom-right corner of the cell's outer frame. Based on the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner of the cell's outer frame, the area of ​​the cell's outer frame can be determined. Furthermore, in "tokens":"A", the "A" represents the text content within the cell corresponding to that "tokens".

[0075] In this embodiment, the server can obtain the names of each field in the first table, the coordinate positions of each field name in the layout coordinate system, and the positions of each table line segment in the first table based on the layout information of the first table corresponding to the sample image. For example, Figure 3As shown, the server can obtain the identifier of the table image corresponding to the first table. When this identifier is: [{"tokens":"Project","bbox":[3,3,165,32]},{"tokens":"Merger","bbox":[166,3,313,17]},{"tokens":"Parent Company","bbox":[315,3,462,17]},{"tokens":"January-March 2004","bbox":[166,18,239,32]},{"tokens":"January-March 2003","bbox" :[240,18,313,32]},{"tokens":"January-March 2004","bbox":[315,18,388,32]},{"tokens":"January-March 2003","bbox":[389,18,462,32]}], From this, we can know that in the first table, the coordinates of the top left corner of cell 1's outer frame are (3,3), the coordinates of the bottom right corner of cell 1's outer frame are (165,32), and the text content inside cell 1 is: Project; the coordinates of the top left corner of cell 2's outer frame are (166,3). Cell 2 has the coordinates of (313, 17) for the bottom right corner of its outer border and contains the text "Merge"; Cell 3 has the coordinates of (315, 3) for the top left corner of its outer border and (462, 17) for the bottom right corner and contains the text "Parent Company"; Cell 4 has the coordinates of (166, 18) for the top left corner of its outer border and (239, 32) for the bottom right corner and contains the text "January-March 2004"; Cell 5 has the coordinates of (313, 17) for the bottom right corner of its outer border and contains the text "January-March 2004"; Cell 5 has the coordinates of (313, 17) for the bottom right corner of its outer border and contains the text "Merge"; Cell 2 ...3, 17) for the bottom right corner of its outer border and contains the text "Merge"; Cell 3 has the coordinates of (315, 3) for the top left corner of its outer border and (462, 17) for the bottom right corner and contains the text "Parent Company"; Cell 4 has the coordinates of (16 The coordinates of the bottom right corner of cell 5 are (313,32), and the text content inside cell 5 is: January-March 2003; The coordinates of the top left corner of cell 6 are (315,18), and the coordinates of the bottom right corner of cell 6 are (388,32), and the text content inside cell 6 is: January-March 2004; The coordinates of the top left corner of cell 7 are (389,18), and the coordinates of the bottom right corner of cell 7 are (462,32), and the text content inside cell 7 is: January-March 2003.

[0076] In this embodiment, the server is based on, as follows Figure 3 The layout information of the first table shown can generate a table with the same field names and the same coordinate positions of each field name in the layout coordinate system, but with different line segments, such as... Figures 4-6The multiple second tables shown.

[0077] In some embodiments, the layout information includes multiple table line segments and the table line segment positions of the multiple table line segments; generating multiple second tables corresponding to the sample image based on the layout information and multiple standard tables may include: for each standard table, obtaining multiple standard line segments of the standard table and the standard line segment positions of the multiple standard line segments; obtaining multiple table line segments and the coordinates of the multiple table line segments positions in the layout information; performing position matching between the multiple table line segment positions and the multiple standard line segment positions, and when there is no standard line segment position that matches the table line segment position, the position matching fails; erasing the table line segments corresponding to the position of the table line segment that failed to match from the first table to obtain the second table corresponding to the sample image.

[0078] In this embodiment, after obtaining the x-coordinate x1 and y-coordinate y1 of the top-left corner of the outer frame of each cell in the first table, the server can also arrange the y-coordinate y1 of the top-left corner of the outer frame of multiple cells in the first table in ascending order. If the y-coordinate y1 of the top-left corner of the outer frame of multiple cells are the same, the x-coordinate x1 of the top-left corner of the outer frame of multiple cells with the same y1 are arranged in ascending order to obtain the arrangement order of multiple cells in the first table.

[0079] In this embodiment, the server can determine the left and top neighbor cells of each cell (target cell) in the first table based on the arrangement order of multiple cells in the first table. The left neighbor cell refers to the cell closest to the target cell among the multiple cells to its left; that is, the cell whose x-coordinate of the top-left corner of its outer frame is less than the x-coordinate of the top-left corner of the target cell's outer frame, and whose x-coordinate is closest to the x-coordinate of the top-left corner of the target cell's outer frame. For example, in the first table, there are cells A1, A2, and A3. The coordinates of the top-left corner of A1 are (x11, y11), the coordinates of the top-left corner of A2 are (x22, y22), and the coordinates of the top-left corner of A3 are (x33, y33). Based on the arrangement order of the cells in the first table, x11 < x22 < x33. When the target cell is A3, the left neighbor cell of A3 is A2.

[0080] Similarly, the upper neighbor cell refers to the cell that is closest to the target cell among multiple cells located above it. Specifically, it's the cell whose top-left corner coordinate is less than the top-left corner coordinate of the target cell's outer frame, and whose top-left corner coordinates are closest to the top-left corner coordinates of the target cell's outer frame. For example, in the first table, there are cells A1, A4, and A5. The top-left corner coordinates of A1 are (x11, y11), A4's are (x44, y44), and A5's are (x55, y55). ​​Based on the order of the cells in the first table, y44 < y55 < y11. When the target cell is A1, the upper neighbor cell of A1 is A5.

[0081] In this embodiment, when it is necessary to erase a table line segment in the first table, since the coordinates of the outer frame of each cell corresponding to each identifier may have pixel deviations, in order to avoid the situation where the deletion is unreasonable due to large deviations, it is necessary to reconstruct the coordinates of the line segment to be erased. For example, when it is necessary to erase the upper line segment of cell A10, the coordinates of the upper left corner of the outer frame of cell A10 given by the identifier are (x10, y10), the coordinates of the lower right corner of the outer frame of cell A10 are (x11, y11), the upper neighbor of A10 is A8, and the coordinates of the upper left corner of the outer frame of cell A8 are (x00, y00), and the coordinates of the lower right corner of the outer frame of cell A9 are (x01, y01). At this time, the area S where the table line segment to be erased is located is as shown in formula (1):

[0082]

[0083] The coordinates of the top-left corner of region S are: The coordinates of the lower right corner of region S are

[0084] Furthermore, the width of the table line segments to be erased is shown in formula (2):

[0085]

[0086] The `int` function rounds down to the nearest integer. For example, `int(1.2)` returns the first integer less than 1.2, which is 1; `int(1.6)` returns 1; and `int(-1.5)` returns -2.

[0087] In some embodiments, erasing the table line segment corresponding to the position of the table line segment where the position matching fails may include: obtaining the table line segment position of the table line segment to be erased as the target object; obtaining the colors of multiple pixels within a preset pixel range corresponding to the target object as intermediate colors; calculating the number of multiple intermediate colors and taking the intermediate color with the most numbers as the target color; and erasing the table line segment to be erased based on the target color.

[0088] In this embodiment, the server needs to determine the pixel color of the table line segment to be erased. To avoid the pixel color used when erasing the table line segment being inconsistent with the table background color, the server can select multiple pixel colors within a preset pixel range corresponding to the table line segment to be erased as intermediate colors; calculate the number of multiple intermediate colors, and use the intermediate color with the most occurrences as the target color; based on the target color, the target object is erased. For example, the server can obtain the pixel colors in region S1, which is 5 pixels to the left of the target object, and the pixel colors in region S2, which is 5 pixels to the right of the target object, as intermediate colors. It is understood that this application does not limit the preset pixel range, and the preset pixel range does not need to satisfy the requirement that there is no text content (such as text, numbers, etc.) in the cell within the preset pixel range corresponding to the target object, in order to avoid the font pixels of the text affecting the erasure of the target object.

[0089] In this embodiment, the server can calculate the most abundant intermediate color among the pixels in regions S1 and S2 as the target color, and then erase the target object based on the target color (i.e., fill the target object with the target color). For example, if regions S1 and S2 contain 100 pixels, of which 50 pixels are white, 20 pixels are black, and 30 pixels are light gray, and 50 > 30 > 20, then the server selects white as the target color to erase the target object.

[0090] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0091] Based on the same inventive concept, this application also provides a sample processing apparatus for implementing the sample processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more sample processing apparatus embodiments provided below can be found in the limitations of the sample processing method described above, and will not be repeated here.

[0092] In one embodiment, such as Figure 8 As shown, a sample processing apparatus is provided, including: a first table acquisition module 802, a second table generation module 804, and a sample dataset expansion module 806, wherein:

[0093] The first table acquisition module 802 is used to input multiple sample images into a pre-trained table matching model to obtain the first table of the sample images.

[0094] The second table generation module 804 is used to generate multiple second tables corresponding to the sample image based on multiple pre-set standard tables and the first table; wherein, the second tables of the sample image and the first table of the sample image constitute multiple standard tables corresponding to the sample image.

[0095] The sample dataset expansion module 806 is used to add the first table and the second table to the sample dataset.

[0096] In one embodiment, the first table acquisition module 802 may include:

[0097] The table matching submodule is used to input multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images and the target field to be identified; the target field belongs to at least one field in the layout information.

[0098] The first table generation submodule is used to generate the first table of the sample image based on the table layout information and the target field to be identified.

[0099] In one embodiment, the table matching submodule may include:

[0100] The layout acquisition unit is used to input multiple sample images into a pre-trained table matching model to obtain the layout of the table in the sample images.

[0101] The field detection unit is used to perform field detection on the layout to determine the names and areas of each field in the layout.

[0102] The layout information determination unit is used to obtain the layout coordinate system corresponding to the layout. Based on the field name and field area, it obtains the coordinate position of each field name in the layout coordinate system and uses each field name and its corresponding coordinate position in the layout coordinate system as layout information.

[0103] In one embodiment, the second table generation module 804 may include:

[0104] The layout information acquisition submodule is used to acquire the layout information of tables in sample images.

[0105] The second table generation submodule is used to generate multiple second tables corresponding to the sample images based on layout information and multiple standard tables.

[0106] In one embodiment, the layout information includes multiple table line segments and the positions of the multiple table line segments; the second table generation submodule may include:

[0107] The standard line segment position acquisition unit is used to acquire multiple standard line segments and the standard line segment positions of multiple standard line segments for each standard table.

[0108] The table position coordinate acquisition unit is used to acquire multiple table line segments and their position coordinates in the layout information.

[0109] The position matching unit is used to match the positions of multiple table line segments with the positions of multiple standard line segments. If there is no standard line segment position that matches the position of the table line segment, the position matching fails.

[0110] The table line segment erasure unit is used to erase the table line segments corresponding to the positions of the table line segments that failed to match from the first table, so as to obtain the second table of the corresponding sample image.

[0111] In one embodiment, the table line erasing unit may include:

[0112] The target object acquisition sub-unit is used to obtain the position of the table line segment to be erased, which is then used as the target object.

[0113] The intermediate color determination subunit is used to obtain the colors of multiple pixels within a preset pixel range corresponding to the target object, and use them as the intermediate color.

[0114] The target color determination subunit is used to calculate the number of multiple intermediate colors and select the intermediate color with the largest number as the target color.

[0115] The erase sub-unit is used to erase the table lines to be erased based on the target color.

[0116] Each module in the aforementioned sample processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0117] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as the x-coordinate of the top-left corner of a cell, the y-coordinate of the top-left corner of a cell, the x-coordinate of the bottom-right corner of a cell, and field names. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a sample processing method.

[0118] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images; generating multiple second tables corresponding to the sample images based on multiple pre-set standard tables and the first table; wherein the second tables of the sample images and the first tables of the sample images constitute multiple standard tables corresponding to the sample images; and adding the first table and the second tables to the sample dataset.

[0120] In one embodiment, when the processor executes the computer program, it further implements inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images. This may include: inputting multiple sample images into the pre-trained table matching model to obtain the layout information of the table in the sample images and the target field to be identified; the target field belongs to at least one field in the layout information; and generating the first table of the sample images based on the layout information of the table and the target field to be identified.

[0121] In one embodiment, when the processor executes the computer program, it further implements inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images. This may include: inputting multiple sample images into a pre-trained table matching model to obtain the layout of the table in the sample images; performing field detection on the layout to determine the name of each field and the area of ​​each field in the layout; obtaining the layout coordinate system corresponding to the layout, and based on the name of each field and the area of ​​each field, obtaining the coordinate position of each field name in the layout coordinate system, and using each field name and the coordinate position of each field name in the layout coordinate system as the layout information.

[0122] In one embodiment, when the processor executes the computer program, it further implements the generation of multiple second tables corresponding to the sample image based on multiple pre-set standard tables and a first table, which may include: obtaining the layout information of the tables in the sample image; and generating multiple second tables corresponding to the sample image based on the layout information and multiple standard tables.

[0123] In one embodiment, the layout information includes multiple table line segments and the table line segment positions of the multiple table line segments; when the processor executes the computer program, it further implements the generation of multiple second tables corresponding to the sample image based on the layout information and multiple standard tables, which may include: for each standard table, obtaining multiple standard line segments of the standard table and the standard line segment positions of the multiple standard line segments; obtaining multiple table line segments and the coordinates of the multiple table line segments positions in the layout information; performing position matching between the multiple table line segment positions and the multiple standard line segment positions, and when there is no standard line segment position that matches the table line segment position, the position matching fails; erasing the table line segment corresponding to the position of the table line segment that failed to match from the first table to obtain the second table corresponding to the sample image.

[0124] In one embodiment, when the processor executes the computer program, it further implements erasing the table line segments corresponding to the positions of the table line segments where the position matching fails. This may include: obtaining the table line segment position of the table line segment to be erased as the target object; obtaining the colors of multiple pixels within a preset pixel range corresponding to the target object as intermediate colors; calculating the number of multiple intermediate colors and taking the intermediate color with the most occurrences as the target color; and erasing the table line segment to be erased based on the target color.

[0125] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When the computer program is executed by a processor, it performs the following steps: inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images; generating multiple second tables corresponding to the sample images based on multiple pre-set standard tables and the first table; wherein the second tables of the sample images and the first tables of the sample images constitute multiple standard tables corresponding to the sample images; and adding the first table and the second tables to the sample dataset.

[0126] In one embodiment, when the computer program is executed by the processor, it further implements inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images, which may include: inputting multiple sample images into the pre-trained table matching model to obtain the layout information of the table in the sample images and the target field to be identified; the target field belongs to at least one field in the layout information; and generating the first table of the sample images based on the layout information of the table and the target field to be identified.

[0127] In one embodiment, when the computer program is executed by the processor, it further implements inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images. This may include: inputting multiple sample images into a pre-trained table matching model to obtain the layout of the table in the sample images; performing field detection on the layout to determine the name of each field and the area of ​​each field in the layout; obtaining the layout coordinate system corresponding to the layout, and based on the name of each field and the area of ​​each field, obtaining the coordinate position of each field name in the layout coordinate system, and using each field name and the coordinate position of each field name in the layout coordinate system as the layout information.

[0128] In one embodiment, when the computer program is executed by the processor, it further implements the generation of multiple second tables corresponding to the sample image based on multiple pre-set standard tables and a first table, which may include: obtaining the layout information of the tables in the sample image; and generating multiple second tables corresponding to the sample image based on the layout information and multiple standard tables.

[0129] In one embodiment, the layout information includes multiple table line segments and the table line segment positions of the multiple table line segments; when the computer program is executed by the processor, it further implements the generation of multiple second tables corresponding to the sample image based on the layout information and multiple standard tables, which may include: for each standard table, obtaining multiple standard line segments of the standard table and the standard line segment positions of the multiple standard line segments; obtaining multiple table line segments and the coordinates of the multiple table line segments positions in the layout information; performing position matching between the multiple table line segment positions and the multiple standard line segment positions, and when there is no standard line segment position that matches the table line segment position, the position matching fails; erasing the table line segment corresponding to the position of the table line segment that failed to match from the first table to obtain the second table corresponding to the sample image.

[0130] In one embodiment, when the computer program is executed by the processor, it further implements erasing the table line segments corresponding to the positions of the table line segments that fail to match the position. This may include: obtaining the table line segment position of the table line segment to be erased as the target object; obtaining the colors of multiple pixels within a preset pixel range corresponding to the target object as intermediate colors; calculating the number of multiple intermediate colors and taking the intermediate color with the most numbers as the target color; and erasing the table line segment to be erased based on the target color.

[0131] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images; generating multiple second tables corresponding to the sample images based on multiple pre-set standard tables and the first table; wherein the second tables of the sample images and the first tables of the sample images constitute multiple standard tables corresponding to the sample images; and adding the first table and the second tables to the sample dataset.

[0132] In one embodiment, when the computer program is executed by the processor, it further implements inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images, which may include: inputting multiple sample images into the pre-trained table matching model to obtain the layout information of the table in the sample images and the target field to be identified; the target field belongs to at least one field in the layout information; and generating the first table of the sample images based on the layout information of the table and the target field to be identified.

[0133] In one embodiment, when the computer program is executed by the processor, it further implements inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the table in the sample images. This may include: inputting multiple sample images into a pre-trained table matching model to obtain the layout of the table in the sample images; performing field detection on the layout to determine the name of each field and the area of ​​each field in the layout; obtaining the layout coordinate system corresponding to the layout, and based on the name of each field and the area of ​​each field, obtaining the coordinate position of each field name in the layout coordinate system, and using each field name and the coordinate position of each field name in the layout coordinate system as the layout information.

[0134] In one embodiment, when the computer program is executed by the processor, it further implements the generation of multiple second tables corresponding to the sample image based on multiple pre-set standard tables and a first table, which may include: obtaining the layout information of the tables in the sample image; and generating multiple second tables corresponding to the sample image based on the layout information and multiple standard tables.

[0135] In one embodiment, the layout information includes multiple table line segments and the table line segment positions of the multiple table line segments; when the computer program is executed by the processor, it further implements the generation of multiple second tables corresponding to the sample image based on the layout information and multiple standard tables, which may include: for each standard table, obtaining multiple standard line segments of the standard table and the standard line segment positions of the multiple standard line segments; obtaining multiple table line segments and the coordinates of the multiple table line segments positions in the layout information; performing position matching between the multiple table line segment positions and the multiple standard line segment positions, and when there is no standard line segment position that matches the table line segment position, the position matching fails; erasing the table line segment corresponding to the position of the table line segment that failed to match from the first table to obtain the second table corresponding to the sample image.

[0136] In one embodiment, when the computer program is executed by the processor, it further implements erasing the table line segments corresponding to the positions of the table line segments that fail to match the position. This may include: obtaining the table line segment position of the table line segment to be erased as the target object; obtaining the colors of multiple pixels within a preset pixel range corresponding to the target object as intermediate colors; calculating the number of multiple intermediate colors and taking the intermediate color with the most numbers as the target color; and erasing the table line segment to be erased based on the target color.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A sample processing method, characterized in that, The method includes: Multiple sample images are input into a pre-trained table matching model to obtain the first table of the sample images; Based on a plurality of pre-set standard tables and the first table, a plurality of second tables corresponding to the sample image are generated; wherein, the second tables of the sample image and the first table of the sample image constitute a plurality of standard tables corresponding to the sample image; the step of generating a plurality of second tables corresponding to the sample image based on a plurality of pre-set standard tables and the first table includes: obtaining layout information of the tables in the sample image; the layout information includes a plurality of table line segments and the table line segment positions of the plurality of table line segments; for each standard table, obtaining a plurality of standard line segments and the standard line segment positions of the plurality of standard line segments; obtaining the coordinates of the plurality of table line segments and the coordinates of ... Add the first table and the second table to the sample dataset.

2. The method according to claim 1, characterized in that, The step of inputting multiple sample images into a pre-trained table matching model to obtain a first table of the sample images includes: Multiple sample images are input into a pre-trained table matching model to obtain the table layout information and the target field to be identified in the sample images; the target field belongs to at least one field in the layout information. Based on the layout information of the table and the target field to be identified, a first table of the sample image is generated.

3. The method according to claim 2, characterized in that, The step of inputting multiple sample images into a pre-trained table matching model to obtain the layout information of the tables in the sample images includes: Multiple sample images are input into a pre-trained table matching model to obtain the layout of the table in the sample images; Perform field detection on the layout to determine the names and areas of each field in the layout; Obtain the layout coordinate system corresponding to the layout. Based on each field name and each field area, obtain the coordinate position of each field name in the layout coordinate system. Use each field name and the coordinate position of each field name in the layout coordinate system as layout information.

4. The method according to claim 1, characterized in that, The step of erasing the table line segment corresponding to the position where the position matching fails includes: Obtain the position of the table line segment to be erased, and use it as the target object; Obtain the colors of multiple pixels within a preset pixel range corresponding to the target object, and use them as the intermediate color; Calculate the number of the multiple intermediate colors, and take the intermediate color with the largest number as the target color; Based on the target color, the table line segments to be erased are erased.

5. A sample processing device, characterized in that, The device includes: The first table acquisition module is used to input multiple sample images into a pre-trained table matching model to obtain the first table of the sample images; The second table generation module is used to generate multiple second tables corresponding to the sample image based on multiple pre-set standard tables and the first table; wherein, the second tables of the sample image and the first table of the sample image constitute multiple standard tables corresponding to the sample image; the second table generation module is specifically used to: obtain the layout information of the tables in the sample image; the layout information includes multiple table line segments and the table line segment positions of the multiple table line segments; for each standard table, obtain multiple standard line segments of the standard table and the standard line segment positions of the multiple standard line segments; obtain the coordinates of multiple table line segments and the multiple table line segment positions in the layout information; perform position matching between the multiple table line segment positions and the multiple standard line segment positions, and when there is no standard line segment position that matches the table line segment position, the position matching fails; erase the table line segment corresponding to the position of the table line segment that failed to match from the first table to obtain the second table corresponding to the sample image; The sample dataset expansion module is used to add the first table and the second table to the sample dataset.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images