A table structure identification method, device and equipment and a storage medium
By acquiring global visual features of table images in the column and row dimensions, the column and row lines of the table are predicted, solving the problem that existing table recognition methods cannot accurately predict table structure, and achieving accurate table structure recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-19
- Publication Date
- 2026-03-31
AI Technical Summary
Existing table recognition methods cannot accurately predict table row and column lines, resulting in an inability to obtain accurate table structure.
By acquiring global visual features of the target table image in the column and row dimensions, the table column lines and row lines are predicted. Based on these features, the table image is segmented into a basic grid, and then the grid is merged to obtain the table structure.
It achieves accurate prediction of table row and column lines, obtains a more accurate table structure, and improves the accuracy of table recognition.
Smart Images
Figure CN116343246B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information recognition technology, and in particular to a method, apparatus, device, and storage medium for recognizing table structures. Background Technology
[0002] In the knowledge and information age, documents are a crucial source of information for many different cognitive processes (such as knowledge database creation, OCR, image understanding, and document retrieval). Automatically processing the information embedded in these documents is essential. Tables, as a special type of entity, are very common in documents. They convey important information in a relatively concise form and are prevalent in many fields such as finance and administration. Therefore, table structure recognition technology has wide applications.
[0003] Most current table structure recognition methods are based on segmentation and merging. That is, firstly, the table row and column lines are predicted based on the table image, then the table image is segmented into a series of basic grids based on the predicted table row and column lines, and finally the basic grids are merged to obtain the table structure corresponding to the table image.
[0004] Understandably, to obtain a more accurate table structure, it is first necessary to predict the table's row and column lines accurately. Most current table recognition methods use semantic segmentation-based prediction to predict table row and column lines. However, semantic segmentation-based prediction methods cannot predict table row and column lines accurately, which leads to the inability to obtain a truly accurate table structure. Summary of the Invention
[0005] In view of this, the present invention provides a table structure recognition method, apparatus, device, and storage medium to solve the problem that existing table recognition methods cannot obtain relatively accurate table structures. The technical solution is as follows:
[0006] A table structure recognition method, comprising:
[0007] Obtain the target table image;
[0008] Obtain the global visual features of the target table image in the column dimension and the global visual features in the row dimension;
[0009] Predict table column lines based on the global visual features of the target table image in the column dimension, and predict table row lines based on the global visual features of the target table image in the row dimension.
[0010] Based on the predicted table column lines and table row lines, the target table image is segmented into a basic grid;
[0011] The basic grids obtained through segmentation are merged to obtain the table structure corresponding to the target table image.
[0012] Optionally, obtaining the global visual features of the target table image in the column dimension and the global visual features in the row dimension includes:
[0013] Visual features are extracted from the target table image to obtain the target visual features;
[0014] The visual features corresponding to each row of pixels in the target table image are integrated, and the integrated row vector is determined as the global visual feature of the target table image in the column dimension.
[0015] The visual features corresponding to each column pixel of the target table image are integrated, and the integrated column vector is determined as the global visual feature of the target table image in the row dimension.
[0016] Optionally, integrating the visual features corresponding to the pixels of each row of the target table image from the target visual features includes:
[0017] The average value of the visual features corresponding to the pixels of each row of the target table image is calculated.
[0018] The step of integrating the visual features corresponding to the pixels of each column of the target table image from the target visual features includes:
[0019] The average value is calculated for the visual features of the target that correspond to the pixels in each column of the table image.
[0020] Optionally, predicting table column lines based on the global visual features of the target table image in the column dimension includes:
[0021] Based on the global visual features of the target table image in the column dimension, predict the first convolution parameters and the region where the table column lines are located;
[0022] The table column lines are determined based on the first convolution parameters, the region where the table column lines are located, and the target visual features;
[0023] The prediction of table row lines based on the global visual features of the target table image in the row dimension includes:
[0024] Based on the global visual features of the target table image in the row dimension, predict the second convolution parameters and the region where the table row lines are located;
[0025] The table row lines are determined based on the second convolution parameters, the region where the table row lines are located, and the target visual features.
[0026] Optionally, determining the table column lines based on the first convolution parameters, the region where the table column lines are located, and the target visual features includes:
[0027] The target visual features are convolved based on the third convolution parameter of the predetermined column dimension to obtain the first visual features;
[0028] Extract the convolution parameters corresponding to the region where the table column lines are located from the first convolution parameters;
[0029] The first visual feature is convolved based on the convolution parameters corresponding to the region where the table column lines are located, and the first convolution operation result is obtained. The table column lines are then determined based on the first convolution operation result.
[0030] The step of determining the table row lines based on the second convolution parameters, the region where the table row lines are located, and the target visual features includes:
[0031] The target visual features are convolved based on the fourth convolution parameter of the predetermined row dimension to obtain the second visual features;
[0032] Extract the convolution parameters corresponding to the area where the table row lines are located from the second convolution parameters;
[0033] The second visual feature is convolved based on the convolution parameters corresponding to the area where the table row lines are located, to obtain the second convolution operation result, and the table row lines are determined based on the second convolution operation result.
[0034] Optionally, the merging process of the basic meshes obtained through segmentation includes:
[0035] Based on the target visual features and the position information of each basic grid, the first feature of each basic grid is obtained;
[0036] Based on the first feature of each basic grid, the basic grids are merged.
[0037] Optionally, obtaining the first feature of each basic grid based on the target visual features and the position information of each basic grid includes:
[0038] Visual features of each base grid are extracted from the target visual features, and positional features of each base grid are obtained based on the positional information of each base grid;
[0039] By fusing the visual and positional features of the same basic grid, the first feature of each basic grid is obtained.
[0040] Optionally, the merging process of each basic grid based on its first feature includes:
[0041] Based on the predetermined fifth convolution parameters, the first features of each basic grid are convolved to obtain the second features of each basic grid.
[0042] For each base grid:
[0043] Based on the first feature of the base grid, predict the convolution parameters corresponding to the base grid;
[0044] Based on the convolution parameters corresponding to the basic grid, a convolution operation is performed on the second feature of each basic grid to obtain the convolution operation result corresponding to the basic grid.
[0045] Based on the convolution operation results corresponding to the base grid, a base grid to be merged with the base grid is determined, and the determined base grid is merged with the base grid.
[0046] A table structure recognition device includes: a table image acquisition module, a global visual feature acquisition module, a table row and column line prediction module, a table image segmentation module, and a table structure determination module;
[0047] The table image acquisition module is used to acquire the target table image;
[0048] The global visual feature acquisition module is used to acquire the global visual features of the target table image in the column dimension and the global visual features in the row dimension.
[0049] The table row and column line prediction module is used to predict table column lines based on the global visual features of the target table image in the column dimension, and to predict table row lines based on the global visual features of the target table image in the row dimension.
[0050] The table image segmentation module is used to segment the target table image into a basic grid based on the predicted table column lines and table row lines.
[0051] The table structure determination module is used to merge the basic grids obtained through segmentation to obtain the table structure corresponding to the target table image.
[0052] A processing device, comprising: a memory and a processor;
[0053] The memory is used to store programs;
[0054] The processor is used to execute the program to implement each step of the table structure recognition method described above.
[0055] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the table structure recognition method described in any of the preceding claims.
[0056] The table structure recognition method provided by this invention, after obtaining a target table image, first extracts visual features from the target table image. After obtaining the target visual features, in order to predict accurate table row and column lines, it further obtains global visual features of the table image in the column dimension and global visual features of the table image in the row dimension based on the target visual features. Then, it predicts table column lines based on the global visual features in the column dimension and table row lines based on the global visual features in the row dimension. Next, it divides the table image into basic grids based on the predicted table column and row lines. Finally, it merges the basic grids to obtain the table structure corresponding to the target table image. Because the table structure recognition method provided by this invention uses the global visual features of the target table image in the row dimension and global visual features of the target table image in the column dimension as the prediction basis when predicting table row and column lines, it can predict relatively accurate table row and column lines. Based on this, the table image division and basic grid merging can obtain a relatively accurate table structure. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0058] Figure 1 This is a schematic diagram of a hardware architecture according to an embodiment of the present invention;
[0059] Figure 2 This is a flowchart illustrating the table structure recognition method provided in an embodiment of the present invention.
[0060] Figure 3 An example of a table image provided in an embodiment of the present invention;
[0061] Figure 4 This is a schematic diagram illustrating the segmentation of a table image into several basic grids based on prediction-based table column lines and table row lines, as provided in an embodiment of the present invention.
[0062] Figure 5 This is a schematic diagram illustrating how the visual features corresponding to each row of pixels in a target table image are integrated into a row vector, as provided in an embodiment of the present invention.
[0063] Figure 6 This is a schematic diagram illustrating how the visual features corresponding to each column of pixels in a target table image are integrated into a column vector, as provided in an embodiment of the present invention.
[0064] Figure 7 This is a schematic diagram of the table column line prediction process provided in an embodiment of the present invention;
[0065] Figure 8 This is a schematic diagram of the table row line prediction process provided in an embodiment of the present invention;
[0066] Figure 9 This is a schematic diagram illustrating the acquisition of the first feature of each basic grid based on the target visual features and the position information of each basic grid, as provided in an embodiment of the present invention.
[0067] Figure 10 This is a schematic diagram illustrating the merging process of each basic grid based on a first feature of each basic grid, as provided in an embodiment of the present invention.
[0068] Figure 11 This is an example of a table structure recognition model provided in an embodiment of the present invention;
[0069] Figure 12 This is a schematic diagram of the table structure recognition device provided in an embodiment of the present invention;
[0070] Figure 13 This is a schematic diagram of the processing device provided in an embodiment of the present invention. Detailed Implementation
[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0072] Currently, most table structure recognition methods based on segmentation and merging adopt semantic segmentation-based prediction to predict table row and column lines during the segmentation stage. The process of predicting table row and column lines based on semantic segmentation involves extracting feature information from the table image using a fully convolutional neural network, and then predicting table row and column lines based on the extracted feature information.
[0073] The inventors of this case studied the prediction method based on semantic segmentation and found that the reason why the prediction accuracy of the prediction method based on semantic segmentation is not high is that the receptive field of the fully convolutional neural network is small, and it can only perceive a small range of feature information (i.e., local feature information).
[0074] Based on the above findings, the inventors continued their research and eventually proposed a more effective table structure recognition method.
[0075] Before introducing the table structure recognition method provided by this invention, the hardware architecture involved in this invention will be described first.
[0076] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.
[0077] For example, electronic device 101 can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as personal computer, laptop, tablet, mobile phone, smart TV, etc.
[0078] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.
[0079] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.
[0080] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired network.
[0081] Electronic device 101 can acquire a target table image (e.g., receive a target table image uploaded by a user), send the target table image to server 102, server 102 identifies the table structure corresponding to the target table image according to the table structure recognition method provided by the present invention, and sends the table structure corresponding to the target table image to electronic device 101 or other devices.
[0082] In another possible implementation, the hardware architecture involved in this invention may include an electronic device. The electronic device is one with strong data processing capabilities.
[0083] For example, an electronic device can be any electronic product that can interact with a user through one or more means such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a personal computer, laptop, tablet, mobile phone, smart TV, etc.
[0084] The electronic device can acquire a target table image (e.g., receive a target table image uploaded by a user) and identify the table structure corresponding to the target table image according to the table structure recognition method provided by this invention.
[0085] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.
[0086] The table structure recognition method provided by the present invention will be described in the following embodiments.
[0087] Please see Figure 2 The diagram illustrates a flowchart of a table structure recognition method provided in an embodiment of the present invention. The method may include:
[0088] Step S201: Obtain the target table image.
[0089] The target table image is the table image to be identified. Figure 3 An example of a target table image is shown.
[0090] Step S202: Obtain the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension.
[0091] Among them, the global visual features of the target table image in the column dimension are the global visual features of each column pixel of the table image, and the global visual features of the target table image in the row dimension are the global visual features of each row pixel of the table image.
[0092] Optionally, when obtaining the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension, the visual features of the target table image can be extracted first to obtain the target visual features, and then the global visual features of the table image in the column dimension and the global visual features of the table image in the row dimension can be obtained based on the target visual features.
[0093] Step S203: Predict table column lines based on the global visual features of the target table image in the column dimension, and predict table row lines based on the global visual features of the target table image in the row dimension.
[0094] This invention uses the global visual features of the target table image in the column dimension as the prediction basis to predict the table column lines, and uses the global visual features of the target table image in the row dimension as the prediction basis to predict the table row lines. This prediction method can predict table column lines and table row lines with relatively high accuracy.
[0095] Step S204: Based on the predicted table column lines and table row lines, segment the target table image into a basic grid.
[0096] Based on the predicted table column lines and table row lines, the target table image can be segmented into several basic grids. Figure 4This diagram illustrates how a target table image is segmented into several basic grids based on predicted table column lines and table row lines.
[0097] Step S205: Merge each basic grid to obtain the table structure corresponding to the target table image.
[0098] When merging the basic grids, the merging status of each basic grid can be determined first, and then the merging process can be performed on each basic grid based on the merging status of each basic grid to obtain the table structure corresponding to the target table image.
[0099] The table structure recognition method provided in this embodiment of the invention, after obtaining a target table image, first extracts visual features from the target table image. After obtaining the target visual features, in order to predict table row and column lines more accurately, it acquires global visual features of the target table image in the column dimension and global visual features of the target table image in the row dimension based on the target visual features. Then, it predicts table column lines based on the global visual features of the target table image in the column dimension and table row lines based on the global visual features of the target table image in the row dimension. Next, it divides the table image into basic grids based on the predicted table column and row lines. Finally, it merges the basic grids to obtain the table structure corresponding to the target table image. Because the table structure recognition method provided in this embodiment of the invention uses the global visual features of the target table image in the row dimension and global visual features of the target table image in the column dimension as the prediction basis when predicting table row and column lines, it can predict table row and column lines more accurately. Based on this, after dividing the table image and merging the basic grids, a more accurate table structure can be obtained.
[0100] In another embodiment of the present invention, the specific implementation process of "step S202: obtaining the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension" in the above embodiment will be described.
[0101] The process of obtaining the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension may include:
[0102] Step a1: Extract visual features from the target table image to obtain the target visual features.
[0103] Among them, the target visual features can characterize the visual information of the target table image.
[0104] Step a2-a: Integrate the visual features corresponding to each row of pixels in the target table image, and determine the integrated row vector as the global visual feature of the target table image in the column dimension.
[0105] Assuming the target table image is H×W×3 (H is the height of the target table image, W is the width of the target table image), visual features are extracted from it to obtain target visual features F, which is a feature of H×W×C (C is the number of feature channels). After obtaining the target visual features F, the visual features corresponding to each row of pixels in F are integrated (i.e., the visual features corresponding to each of the H rows of pixels are integrated), such as... Figure 5 As shown, the 1×W×C features can be obtained, that is, the global visual features G of the table image in the column dimension. col .
[0106] Optionally, the method for integrating the visual features corresponding to each row of pixels in the target table image can be to calculate the average of the visual features corresponding to each row of pixels in the target table image. Of course, this invention is not limited to this; other methods that can integrate the visual features corresponding to each row of pixels in the target table image to obtain the global visual features of the table image in the column dimension are also applicable to this invention.
[0107] Steps a2-b: Integrate the visual features of the target visual features corresponding to the pixels of each column of the table image, and determine the integrated column vector as the global visual feature of the target table image in the row dimension.
[0108] Assuming the target table image is H×W×3 (H is the image height, W is the image width), visual features are extracted to obtain target visual features F, which is a feature set of H×W×C (C is the number of feature channels). After obtaining the target visual features F, the visual features corresponding to each column of pixels in F are integrated (i.e., the visual features corresponding to each of the W columns of pixels are integrated), such as... Figure 6 As shown, the features of H×1×C can be obtained, that is, the global visual features G of the table image in the row dimension. row .
[0109] Optionally, the method for integrating the visual features corresponding to each column of the table image in the target visual features can be to calculate the average of the visual features corresponding to each column of the target table image. Of course, the present invention is not limited to this, and other methods that can integrate the visual features corresponding to each column of the target table image to obtain the global visual features of the table image in the row dimension are also applicable to the present invention.
[0110] The above process can be used to obtain the global visual features of the target table image in the column dimension and the global visual features in the row dimension.
[0111] In another embodiment of the present invention, the specific implementation process of "step S203: predicting table column lines based on global visual features of the target table image in the column dimension, and predicting table row lines based on global visual features of the target table image in the row dimension" in the above embodiment will be described.
[0112] The process of predicting table column lines based on global visual features of the table image in the column dimension can include:
[0113] Step b1: Based on the global visual features of the target table image in the column dimension, predict the first convolution parameters and the region where the table column lines are located.
[0114] like Figure 7 As shown, the global visual features G of the target table image in the column dimension are obtained. col Afterwards, it can be based on G col Predict the first convolution parameter θ col and the area P where the table column lines are located col Among them, the global visual features G of the target table image in the column dimension are used. col The predicted first convolution parameter θ col This includes the convolution parameters corresponding to the pixels in each column of the target table image.
[0115] Step b2: Determine the table column lines based on the first convolution parameters, the region where the table column lines are located, and the target visual features.
[0116] Specifically, the process of determining table column lines based on the first convolution parameters, the region where the table column lines are located, and the target visual features can include:
[0117] Step b21: Perform a convolution operation on the target visual features based on the predetermined third convolution parameters of the column dimensions to obtain the first visual features.
[0118] like Figure 7 As shown, by performing a convolution operation on the target visual feature F based on the predetermined third convolution parameters of the column dimension, the first visual feature F can be obtained. col First visual feature F col It has the same size as the target visual feature F.
[0119] Step b22: Extract the convolution parameters corresponding to the region where the table column lines are located from the first convolution parameters.
[0120] From the first convolution parameter θ col The convolution parameters corresponding to the region where each column line of the table is located are extracted. As mentioned above, the first convolution parameter includes the convolution parameters corresponding to each column of pixels in the target table image. If the region where a column of pixels is located is the region where a column line of the table is located, then the convolution parameters corresponding to that column of pixels are used as the convolution parameters corresponding to the region where the column line of the table is located.
[0121] Step b23: Perform a convolution operation on the first visual feature based on the convolution parameters corresponding to the area where the table column lines are located, obtain the first convolution operation result, and determine the table column lines based on the first convolution operation result.
[0122] For each table column line region, after obtaining the convolution parameters corresponding to that region, the first visual feature F is applied based on those convolution parameters. col A convolution operation is performed to obtain the first convolution result corresponding to the region where the table column line is located. Then, the table column line is determined based on the first convolution result corresponding to the region where the table column line is located. Finally, a table column line is determined for each region where the table column line is located.
[0123] The embodiments of the present invention determine the table column lines based on dynamic convolution, that is, dynamically predict the first convolution parameters, perform convolution operation on the first visual features based on the predicted first convolution parameters, and then determine the table column lines based on the convolution operation results.
[0124] The process of predicting table row lines based on global visual features in the row dimension of the target table image can include:
[0125] Step c1: Based on the global visual features of the table image in the row dimension, predict the second convolution parameters and the region where the table row lines are located.
[0126] like Figure 8 As shown, the global visual features G of the target table image in the row dimension are obtained. row Afterwards, it can be based on G row Predict the second convolution parameter θ row and the area P where the table row lines are located row Among them, the global visual features G of the target table image in the row dimension are used. row Predicted second convolution parameter θ col This includes the convolution parameters corresponding to the pixels of each row in the target table image.
[0127] Step c2: Determine the table row lines based on the second convolution parameters, the area where the table row lines are located, and the target visual features.
[0128] Specifically, the process of determining table row lines based on the second convolution parameters, the region where the table row lines are located, and the target visual features can include:
[0129] Step c21: Perform a convolution operation on the target visual features based on the predetermined fourth convolution parameters of the row dimension to obtain the second visual features.
[0130] like Figure 8As shown, by performing a convolution operation on the target visual feature F based on the predetermined fourth convolution parameters of the row dimension, the second visual feature F can be obtained. row Second visual feature F row It has the same size as the target visual feature F.
[0131] Step c22: Extract the convolution parameters corresponding to the area where the table row lines are located from the second convolution parameters.
[0132] From the second convolution parameter θ row The convolution parameters corresponding to the region where each table row line is located are extracted. As mentioned above, the second convolution parameter includes the convolution parameters corresponding to each row of pixels in the target table image. If the region where a row of pixels is located is the region where a table row line is located, then the convolution parameters corresponding to that row of pixels are used as the convolution parameters corresponding to that region where the table row line is located.
[0133] Step c23: Perform a convolution operation on the second visual features based on the convolution parameters corresponding to the area where the table row lines are located, obtain the result of the second convolution operation, and determine the table row lines based on the result of the second convolution operation.
[0134] For each table row line region, after obtaining the convolution parameters corresponding to that region, the second visual feature F is applied based on those convolution parameters. row A convolution operation is performed to obtain the result of the second convolution operation corresponding to the region where the table row line is located. Then, the table row line is determined based on the result of the second convolution operation corresponding to the region where the table row line is located. Finally, one table row line can be determined for each region where the table row line is located.
[0135] The embodiments of the present invention determine the table row lines based on dynamic convolution, that is, dynamically predict the second convolution parameters, perform convolution operation on the second visual features based on the predicted second convolution parameters, and then determine the table row lines based on the convolution operation results.
[0136] This invention predicts the location of table row lines based on global visual features of the target table image in the row dimension, and predicts the location of table column lines based on global visual features of the target table image in the column dimension. Furthermore, it predicts table row and column lines using dynamic convolution. Because the table row and column line prediction method provided by this invention uses global visual features as the prediction basis, it can predict relatively accurate regions. Based on this, dynamic convolution can predict relatively accurate table row and column lines. In other words, the table row and column line prediction method provided by this invention can predict relatively accurate table row and column lines and is less susceptible to interference from complex background noise.
[0137] In another embodiment of the present invention, the specific implementation process of "step S205: merging each basic grid to obtain the table structure corresponding to the target table image" in the above embodiment will be described.
[0138] The process of merging the basic grids to obtain the table structure corresponding to the target table image may include:
[0139] Step d1: Based on the target visual features and the position information of each basic grid, obtain the first feature of each basic grid.
[0140] Among them, the position information of a basic grid is information that can indicate the position of the basic grid in the target table image.
[0141] Specifically, the process of obtaining the first feature of each basic grid based on the target visual features and the positional information of each basic grid includes: firstly, extracting the visual features of each basic grid from the target visual features, and then obtaining the positional features of each basic grid based on its positional information; finally, fusing the visual features and positional features of the same basic grid to obtain the first feature of each basic grid. Specifically, when extracting the visual features of each basic grid from the target visual features, for each basic grid, the visual features of that basic grid are extracted from the target visual features based on its positional information.
[0142] For example, the target table image is segmented into 16 basic grids based on table row lines and table column lines. See [link to relevant documentation]. Figure 9 The diagram illustrates how to obtain the first features of each basic grid based on the target visual features and the position information of each basic grid. First, based on the position information of the 16 basic grids, the visual features of the 16 basic grids are extracted from the target visual features F. At the same time, the position information of the 16 basic grids is encoded to obtain the position features of the 16 basic grids. Then, the visual features and position information of the same basic grid are fused to obtain the first features of the 16 basic grids.
[0143] Step d2: Based on the first feature of each basic grid, merge the basic grids.
[0144] Specifically, the process of merging the basic grids based on their first feature can include:
[0145] Step d21: Perform convolution operation on the first feature of each basic grid based on the predetermined fifth convolution parameters to obtain the second feature of each basic grid.
[0146] For example, the target table image is segmented into 16 basic grids based on table row lines and table column lines, such as... Figure 10As shown, the first features of the 16 basic grids are convolved based on the predetermined fifth convolution parameters to obtain the second features of the 16 basic grids.
[0147] It should be noted that the purpose of performing convolution operations on the first feature of each basic grid is to make the feature more compatible with subsequent dynamic convolution operations. The second feature of a basic grid has the same size as the first feature of that basic grid.
[0148] Step d22: For each base grid, based on the first feature of the base grid, predict the convolution parameters corresponding to the base grid. Based on the convolution parameters corresponding to the base grid, perform a convolution operation on the second feature of each base grid to obtain the convolution operation result corresponding to the base grid. Based on the convolution operation result corresponding to the base grid, determine the base grid to be merged with the base grid, and merge the determined base grid with the base grid.
[0149] For example, the target table image is segmented into 16 basic grids based on table row lines and table column lines, such as... Figure 10 As shown, for the first basic grid grid11 in the first row, the convolution parameters corresponding to grid11 are first predicted based on the first feature of grid11. Then, a convolution operation is performed on the second feature of grid11 based on the convolution parameters, resulting in a score map corresponding to grid11. The score map contains the merge relationship scores between grid11 and each basic grid (i.e., 16 scores). Scores greater than a set score threshold are selected from these 16 scores. The basic grids whose merge relationship scores with grid11 are greater than the set score threshold are determined as the basic grids to be merged with grid11. Figure 10 As shown, the first basic grid in the second row, grid21, is the basic grid merged with grid11. Grid21 is merged with grid11. For the second basic grid in the first row, grid12, firstly, the convolution parameters corresponding to grid12 are predicted based on the first feature of grid12. Then, a convolution operation is performed on the second feature of grid12 based on the convolution parameters, resulting in a score map corresponding to grid12. The score map corresponding to grid12 contains the merge relationship scores between grid2 and each basic grid (i.e., 16 scores). From these 16 scores, scores greater than a set score threshold are selected. Basic grids whose merge relationship scores with grid12 are greater than the set score threshold are determined as the basic grids merged with grid12. Figure 10As shown, the third and fourth basic grids in the first row, grid13 and grid14, are the basic grids merged with grid12. By merging grid13 and grid14 with grid12 and processing the other basic grids in the same way, the table structure corresponding to the target table image can be obtained.
[0150] It should be noted that the merging process of each basic grid can be executed in parallel. That is, the basic grids to be merged with each basic grid can be determined and merged in parallel. In other words, the present invention can merge the basic grids in parallel based on dynamic convolution, which can greatly improve the efficiency of basic grid merging and thus quickly restore the grid structure.
[0151] In one possible implementation, steps S201 to S205 described above can be implemented based on a model. Of course, the present invention is not limited to this, that is, the present invention does not limit the specific implementation of steps S201 to S205.
[0152] Please see Figure 11 This illustrates an example of a table recognition model capable of recognizing table structure. The following section will discuss models based on... Figure 11 The process of table structure recognition using the table recognition model shown is introduced.
[0153] like Figure 11 As shown, the table recognition model may include a visual feature extraction module 1101, a table image segmentation module 1102, a basic grid feature acquisition module 1103, and a basic grid merging module 1104, based on... Figure 11 The table recognition model shown illustrates the process of recognizing table structure, which includes:
[0154] Step e1: Obtain the target table image.
[0155] Step e2: Input the target table image into the visual feature extraction module 1101 of the table recognition model. The visual feature extraction module 1101 extracts visual features from the target table image to obtain the target visual features.
[0156] Step e3: Input the target visual features into the table image segmentation module 1102. The table image segmentation module 1102 predicts the table row lines and table column lines based on the target visual features, and segments the target table image into a basic grid based on the predicted table column lines and table row lines.
[0157] Specifically, the table image segmentation module 1102 may include a table column line prediction module and a table row line prediction module. The target visual features are input into the table column line prediction module to predict the table column lines, and the target visual features are input into the table row line prediction module to predict the table row lines.
[0158] The table column line prediction module performs the following process: on the one hand, it obtains the global visual features of the target table image in the column dimension based on the target visual features, and predicts the first convolution parameters and the area where the table column lines are located based on the global visual features of the target table image in the column dimension; on the other hand, it performs a convolution operation on the target visual features based on the predetermined third convolution parameters in the column dimension to obtain the first visual features. After obtaining the first visual features, the first convolution parameters, and the area where the table column lines are located, it first extracts the convolution parameters corresponding to the area where the table column lines are located from the first convolution parameters, and then performs a convolution operation on the first visual features based on the convolution parameters corresponding to the area where the table column lines are located to obtain the result of the first convolution operation. The table column lines are determined based on the result of the first convolution operation.
[0159] The table row line prediction module performs the following process: First, it acquires the global visual features of the target table image in the row dimension based on the target visual features, and predicts the second convolution parameters and the area where the table row lines are located based on the global visual features of the target table image in the row dimension. Second, it performs a convolution operation on the target visual features based on the predetermined fourth convolution parameters in the row dimension to obtain the second visual features. After obtaining the second visual features, the second convolution parameters, and the area where the table row lines are located, it first extracts the convolution parameters corresponding to the area where the table row lines are located from the second convolution parameters, and then performs a convolution operation on the second visual features based on the convolution parameters corresponding to the area where the table row lines are located to obtain the result of the second convolution operation. The table row lines are determined based on the result of the second convolution operation.
[0160] Step e4: Input the target visual features and the position information of each basic grid into the table recognition model's basic grid feature acquisition module 1103. The basic grid feature acquisition module 1103 acquires the first feature of each basic grid based on the target visual features and the position information of each basic grid.
[0161] Specifically, the process by which the basic grid feature acquisition module 1103 acquires the first features of each basic grid based on the target visual features and the positional information of each basic grid includes: acquiring the first features of each basic grid based on the target visual features and the positional information of each basic grid. More specifically, the basic grid feature acquisition module 1103 first extracts the visual features of each basic grid from the target visual features, and acquires the positional features of each basic grid based on the positional information of each basic grid. Then, it fuses the visual features and positional features of the same basic grid to obtain the first features of each basic grid.
[0162] Step e5: Input the first features of each basic grid into the table to identify the basic grid merging module 1104 of the model. The basic grid merging module 1104 performs merging processing on each basic grid based on the first features of each basic grid.
[0163] Specifically, the basic grid merging module 1104 performs the merging process on each basic grid based on its first feature, including: performing a convolution operation on the first feature of each basic grid based on a pre-determined fifth convolution parameter to obtain the second feature of each basic grid; for each basic grid: predicting the convolution parameter corresponding to the basic grid based on its first feature; performing a convolution operation on the second feature of each basic grid based on the convolution parameter corresponding to the basic grid to obtain the convolution operation result corresponding to the basic grid; determining the basic grid to be merged with the basic grid based on the convolution operation result corresponding to the basic grid; and merging the determined basic grid with the basic grid.
[0164] For more detailed implementation processes and related explanations of each step in this embodiment, please refer to the relevant parts of the above embodiments. This embodiment will not repeat them here.
[0165] In this embodiment, the table structure recognition model is trained using multiple training table images labeled with table structure information. During training, the training table images are first input into the model to obtain the corresponding table structure recognition results. Then, based on the recognition results and the labeled table structure information, the prediction loss of the model is determined. Finally, the model's parameters are updated according to the prediction loss. This process is repeated multiple times until the training termination condition is met (e.g., model convergence, or reaching a preset number of training iterations).
[0166] This invention also provides a table structure recognition device. The table structure recognition device provided in this invention will be described below. The table structure recognition device described below can be referred to in correspondence with the table structure recognition method described above.
[0167] Please see Figure 12 The diagram shows a schematic of the table structure recognition device provided in an embodiment of the present invention. The table structure recognition device may include: a table image acquisition module 1201, a global visual feature acquisition module 1202, a table row and column line prediction module 1203, a table image processing module 1204, and a table structure determination module 1205.
[0168] The table image acquisition module 1201 is used to acquire the target table image.
[0169] The global visual feature acquisition module 1202 is used to acquire the global visual features of the target table image in the column dimension and the global visual features in the row dimension.
[0170] The table row and column line prediction module 1203 is used to predict table column lines based on the global visual features of the target table image in the column dimension, and to predict table row lines based on the global visual features of the target table image in the row dimension.
[0171] The table image processing module 1204 is used to segment the target table image into a basic grid based on the predicted table column lines and table row lines.
[0172] The table structure determination module 1205 is used to merge the basic grids obtained through segmentation to obtain the table structure corresponding to the target table image.
[0173] Optionally, when acquiring the global visual feature acquisition module 1202 of the target table image in the column dimension and the global visual feature in the row dimension, it is specifically used for:
[0174] Visual features are extracted from the target table image to obtain the target visual features;
[0175] The visual features corresponding to each row of pixels in the target table image are integrated, and the integrated row vector is determined as the global visual feature of the target table image in the column dimension.
[0176] The visual features corresponding to each column pixel of the target table image are integrated, and the integrated column vector is determined as the global visual feature of the target table image in the row dimension.
[0177] Optionally, when integrating the visual features corresponding to each row of pixels in the target table image, the global visual feature acquisition module 1202 is specifically used for:
[0178] The average value of the visual features corresponding to the pixels of each row of the target table image is calculated.
[0179] Optionally, when integrating the visual features corresponding to the pixels of each column of the target table image, the global visual feature acquisition module 1202 is specifically used for:
[0180] The average value is calculated for the visual features of the target that correspond to the pixels in each column of the table image.
[0181] Optionally, when predicting table column lines based on the global visual features of the target table image in the column dimension, the table row and column line prediction module 1203 is specifically used for:
[0182] Based on the global visual features of the target table image in the column dimension, predict the first convolution parameters and the region where the table column lines are located;
[0183] The table column lines are determined based on the first convolution parameters, the region where the table column lines are located, and the target visual features;
[0184] Optionally, when predicting table row and column lines based on the global visual features of the target table image in the row dimension, the table row and column line prediction module 1203 is specifically used for:
[0185] Based on the global visual features of the target table image in the row dimension, predict the second convolution parameters and the region where the table row lines are located;
[0186] The table row lines are determined based on the second convolution parameters, the region where the table row lines are located, and the target visual features.
[0187] Optionally, when determining table row and column line prediction modules 1203 based on the first convolution parameters, the region where the table column lines are located, and the target visual features, they are specifically used for:
[0188] The target visual features are convolved based on the third convolution parameter of the predetermined column dimension to obtain the first visual features;
[0189] Extract the convolution parameters corresponding to the region where the table column lines are located from the first convolution parameters;
[0190] The first visual feature is convolved based on the convolution parameters corresponding to the region where the table column lines are located, and the first convolution operation result is obtained. The table column lines are then determined based on the first convolution operation result.
[0191] Optionally, when determining table row and column line prediction modules 1203 based on the second convolution parameters, the region where the table row lines are located, and the target visual features, the module is specifically used for:
[0192] The target visual features are convolved based on the fourth convolution parameter of the predetermined row dimension to obtain the second visual features;
[0193] Extract the convolution parameters corresponding to the area where the table row lines are located from the second convolution parameters;
[0194] The second visual feature is convolved based on the convolution parameters corresponding to the area where the table row lines are located, to obtain the second convolution operation result, and the table row lines are determined based on the second convolution operation result.
[0195] Optionally, the table structure determination module 1205, when merging the basic meshes obtained through segmentation, is specifically used for:
[0196] Based on the target visual features and the position information of each basic grid, the first feature of each basic grid is obtained;
[0197] Based on the first feature of each basic grid, the basic grids are merged.
[0198] Optionally, when the table structure determination module 1205 obtains the first feature of each basic grid based on the target visual features and the position information of each basic grid, it is specifically used for:
[0199] Visual features of each base grid are extracted from the target visual features, and positional features of each base grid are obtained based on the positional information of each base grid;
[0200] By fusing the visual and positional features of the same basic grid, the first feature of each basic grid is obtained.
[0201] Optionally, when the table structure determination module 1205 performs merging processing on each basic grid based on the first feature of each basic grid, it is specifically used for:
[0202] Based on the predetermined fifth convolution parameters, the first features of each basic grid are convolved to obtain the second features of each basic grid.
[0203] For each base grid:
[0204] Based on the first feature of the base grid, predict the convolution parameters corresponding to the base grid;
[0205] Based on the convolution parameters corresponding to the basic grid, a convolution operation is performed on the second feature of each basic grid to obtain the convolution operation result corresponding to the basic grid.
[0206] Based on the convolution operation results corresponding to the base grid, a base grid to be merged with the base grid is determined, and the determined base grid is merged with the base grid.
[0207] The table structure recognition device provided in this invention, after obtaining a target table image, first extracts visual features from the target table image. After obtaining the target visual features, in order to predict accurate table row and column lines, it acquires global visual features of the table image in the column dimension and global visual features of the table image in the row dimension based on the target visual features. Then, it predicts table column lines based on the global visual features in the column dimension and table row lines based on the global visual features in the row dimension. Next, it divides the table image into basic grids based on the predicted table column and row lines. Finally, it merges the basic grids to obtain the table structure corresponding to the target table image. Because the table structure recognition device provided in this invention uses the global visual features of the table image in the row dimension and global visual features of the table image in the row dimension as the prediction basis when predicting table row and column lines, it can predict relatively accurate table row and column lines. Based on this, dividing the table image and merging the basic grids can obtain a relatively accurate table structure.
[0208] This invention also provides a processing device; please refer to [link / reference]. Figure 13 The diagram shows the structure of the processing device, which may include: a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304.
[0209] In this embodiment of the invention, the number of processor 1301, communication interface 1302, memory 1303 and communication bus 1304 is at least one, and processor 1301, communication interface 1302 and memory 1303 communicate with each other through communication bus 1304.
[0210] The processor 1301 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0211] The memory 1303 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0212] The memory stores a program, which the processor can call. The program is used for:
[0213] Obtain the target table image;
[0214] Obtain the global visual features of the target table image in the column dimension and the global visual features in the row dimension;
[0215] Predict table column lines based on the global visual features of the target table image in the column dimension, and predict table row lines based on the global visual features of the target table image in the row dimension.
[0216] Based on the predicted table column lines and table row lines, the target table image is segmented into a basic grid;
[0217] The basic grids obtained through segmentation are merged to obtain the table structure corresponding to the target table image.
[0218] Optionally, the refined and extended functions of the program can be found in the description above.
[0219] This invention also provides a readable storage medium that stores a program suitable for execution by a processor, the program being used for:
[0220] Obtain the target table image;
[0221] Obtain the global visual features of the target table image in the column dimension and the global visual features in the row dimension;
[0222] Predict table column lines based on the global visual features of the target table image in the column dimension, and predict table row lines based on the global visual features of the target table image in the row dimension.
[0223] Based on the predicted table column lines and table row lines, the target table image is segmented into a basic grid;
[0224] The basic grids obtained through segmentation are merged to obtain the table structure corresponding to the target table image.
[0225] Optionally, the refined and extended functions of the program can be found in the description above.
[0226] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0227] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0228] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A table structure recognition method characterized by comprising: The method comprises: obtaining a target table image; obtaining global visual features of the target table image in the column dimension and global visual features of the target table image in the row dimension; predicting table column lines based on the global visual features of the target table image in the column dimension and predicting table row lines based on the global visual features of the target table image in the row dimension; segmenting the target table image into basic grids based on the predicted table column lines and table row lines; performing merging processing on each basic grid obtained by segmentation to obtain a table structure corresponding to the target table image; wherein the obtaining of the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension comprises: extracting visual features of the target table image to obtain target visual features; obtaining the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension based on the target visual features.
2. The table structure recognition method according to claim 1, characterized in that, The obtaining of the global visual features of the target table image in the column dimension and the global visual features of the target table image in the row dimension based on the target visual features comprises: integrating visual features corresponding to each row of pixels of the target table image in the target visual features, and determining a row vector obtained by the integration as the global visual features of the target table image in the column dimension; integrating visual features corresponding to each column of pixels of the target table image in the target visual features, and determining a column vector obtained by the integration as the global visual features of the target table image in the row dimension.
3. The table structure recognition method according to claim 2, characterized in that, The integration of the visual features corresponding to each row of pixels of the target table image in the target visual features comprises: averaging the visual features corresponding to each row of pixels of the target table image in the target visual features. The integration of the visual features corresponding to each column of pixels of the target table image in the target visual features comprises: averaging the visual features corresponding to each column of pixels of the target table image in the target visual features.
4. The table structure recognition method according to claim 1, characterized by, The prediction of the table column lines based on the global visual features of the target table image in the column dimension comprises: predicting a first convolution parameter and a region where the table column lines are located based on the global visual features of the target table image in the column dimension; determining the table column lines based on the first convolution parameter, the region where the table column lines are located, and the target visual features. The prediction of the table row lines based on the global visual features of the target table image in the row dimension comprises: predicting a second convolution parameter and a region where the table row lines are located based on the global visual features of the target table image in the row dimension; determining the table row lines based on the second convolution parameter, the region where the table row lines are located, and the target visual features.
5. The table structure recognition method according to claim 4, characterized in that, The determination of the table column lines based on the first convolution parameter, the region where the table column lines are located, and the target visual features comprises: performing convolution operation on the target visual features based on a third convolution parameter in the column dimension to obtain first visual features; extracting a convolution parameter corresponding to the region where the table column lines are located from the first convolution parameter; perform convolution operation on the first visual feature based on the convolution parameter corresponding to the region where the table column line is located to obtain a first convolution operation result, and determine the table column line based on the first convolution operation result; the determining the table row line based on the second convolution parameter, the region where the table row line is located, and the target visual feature comprises: performing convolution operation on the target visual feature based on a fourth convolution parameter of a predetermined row dimension to obtain a second visual feature; extracting the convolution parameter corresponding to the region where the table row line is located from the second convolution parameter; performing convolution operation on the second visual feature based on the convolution parameter corresponding to the region where the table row line is located to obtain a second convolution operation result, and determining the table row line based on the second convolution operation result.
6. The table structure recognition method according to any one of claims 1 to 5, characterized in that, the merging processing on each basic grid obtained by the segmentation comprises: obtaining a first feature of each basic grid based on the target visual feature and the position information of each basic grid; performing merging processing on each basic grid based on the first feature of each basic grid.
7. The table structure recognition method according to claim 6, characterized in that, the obtaining the first feature of each basic grid based on the target visual feature and the position information of each basic grid comprises: extracting a visual feature of each basic grid from the target visual feature, and obtaining a position feature of each basic grid based on the position information of each basic grid; fusing the visual feature and the position feature of the same basic grid to obtain the first feature of each basic grid.
8. The table structure recognition method according to claim 6, characterized in that, the performing merging processing on each basic grid based on the first feature of each basic grid comprises: performing convolution operation on the first feature of each basic grid based on a fifth convolution parameter to obtain a second feature of each basic grid; for each basic grid: predicting a convolution parameter corresponding to the basic grid based on the first feature of the basic grid; performing convolution operation on the second feature of each basic grid based on the convolution parameter corresponding to the basic grid to obtain a convolution operation result corresponding to the basic grid; determining a basic grid to be merged with the basic grid based on the convolution operation result corresponding to the basic grid, and merging the determined basic grid with the basic grid.
9. A table structure recognizing apparatus characterized by comprising: comprise: a table image acquisition module, a global visual feature acquisition module, a table row and column line prediction module, a table image segmentation module, and a table structure determination module; the table image acquisition module is configured to acquire a target table image; the global visual feature acquisition module is configured to acquire a global visual feature of the target table image in a column dimension and a global visual feature of the target table image in a row dimension; the table row and column line prediction module is configured to predict a table column line based on the global visual feature of the target table image in the column dimension, and predict a table row line based on the global visual feature of the target table image in the row dimension; the table image segmentation module is configured to segment the target table image into basic grids based on the predicted table column line and table row line; the table structure determination module is configured to perform merging processing on each basic grid obtained by the segmentation to obtain a table structure corresponding to the target table image. In the global visual feature acquisition module, when acquiring the global visual feature of the target table image in the column dimension and the global visual feature of the target table image in the row dimension, the global visual feature acquisition module is specifically configured to: extract a visual feature of the target table image to obtain a target visual feature; acquire the global visual feature of the target table image in the column dimension and the global visual feature of the target table image in the row dimension based on the target visual feature.
10. A processing device, characterized by comprise: a memory and a processor; the memory is configured to store a program; the processor is configured to execute the program to implement each step of the table structure identification method according to any one of claims 1-8.
11. A readable storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement each step of the table structure identification method according to any one of claims 1-8.