Table image processing method, related equipment and readable storage medium

Abstract

The present application discloses a table image processing method, related equipment and readable storage medium. After obtaining a table image to be processed, the table image is first segmented to obtain the basic grid units corresponding to the table image. Then, the visual features of each basic grid unit and the text semantic features of each basic grid unit are determined. Then, based on the visual features of each basic grid unit and the text semantic features of each basic grid unit, the structural information of each basic grid unit is determined. Finally, according to the structural information of each basic grid unit, a table corresponding to the table image is generated. In this solution, when converting a table image into a table, the visual information and text semantic information of the table image are considered at the same time. Compared with converting a table using only the visual information of the table image, the accuracy of the converted table is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a table image processing method, related equipment and readable storage medium. Background Art

[0002] Tables are a common way to display data, usually used to represent structured data. In many scenarios (such as taking photos and scanning pictures), tables exist in the form of images. Converting table images into tables that can be stored in a structured manner is the basis for obtaining key and valuable information from table images.

[0003] Currently, in the process of converting a table image into a table, only the visual information of the table image is used, resulting in low accuracy of the converted table.

[0004] Therefore, in the process of converting a table image into a table, how to improve the accuracy of the converted table becomes a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, this application proposes a table image processing method, related equipment and readable storage medium. The specific scheme is as follows:

[0006] A table image processing method, the method comprising:

[0007] Get the table image to be processed;

[0008] Segmenting the table image to obtain basic grid units corresponding to the table image;

[0009] For each basic grid unit, determining a visual feature of the basic grid unit and a text semantic feature of each basic grid unit;

[0010] Determine structural information of each basic grid unit based on visual features of each basic grid unit and text semantic features of each basic grid unit;

[0011] A table corresponding to the table image is generated according to the structural information of each basic grid unit.

[0012] Optionally, segmenting the table image to obtain basic grid units corresponding to the table image includes:

[0013] Extracting visual features of the table image to obtain visual features of the table image;

[0014] Determining table line position information in the table image based on visual features of the table image;

[0015] Based on the table line position information in the table image, the table image is segmented to obtain basic grid units corresponding to the table image.

[0016] Optionally, determining the visual features of the basic grid unit includes:

[0017] Acquire visual features of the table image;

[0018] The visual features of the basic grid unit are extracted from the visual features of the table image.

[0019] Optionally, determining the text semantic features of the basic grid unit includes:

[0020] Identifying text content corresponding to the basic grid unit;

[0021] Semantic features are extracted from the text content to obtain text semantic features of the basic grid unit.

[0022] Optionally, determining the structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit includes:

[0023] Determine the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit;

[0024] Based on the global characteristics of each basic grid unit, the structural information of each basic grid unit is determined.

[0025] Optionally, determining the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit includes:

[0026] For each basic grid unit, the visual features and text semantic features of the basic grid unit are fused to obtain the initial global features of the basic grid unit;

[0027] Based on the mutual dependence between the initial global features of each basic grid unit, the initial global features of each basic grid unit are adjusted to obtain the adjusted global features of each basic grid unit;

[0028] The adjusted global features of each basic grid unit are used as the global features of the basic grid unit.

[0029] Optionally, determining the structural information of each basic grid unit based on the global features of each basic grid unit includes:

[0030] Based on the global features of each basic grid unit, each basic grid unit is decoded to determine the structural information of each basic grid unit. The structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit.

[0031] Optionally, decoding each basic grid unit based on the global features of each basic grid unit to determine the structural information of each basic grid unit includes:

[0032] The global features of each basic grid unit are input into a decoding network, and the decoding network decodes each basic grid unit in sequence from left to right and from top to bottom to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, as well as the upper node and left node of the basic grid unit.

[0033] Optionally, for each basic grid unit, the decoding network decodes the basic grid unit in the following manner:

[0034] Decoding the global features of the basic grid unit to obtain the decoded features of the basic grid unit and the category of the basic grid unit, where the category is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image;

[0035] Get the decoded features of the decoded basic grid cells;

[0036] Using the decoding feature of the basic grid unit as a query and the decoding feature of the decoded basic grid unit as a key, determining the upper decoding feature and the left decoding feature of the basic grid unit from the decoding features of the decoded basic grid unit;

[0037] Based on the upper decoding feature and the left decoding feature of the basic grid unit, an upper node and a left node of the basic grid unit are determined.

[0038] A table image processing device, the device comprising:

[0039] An acquisition unit, used for acquiring a table image to be processed;

[0040] A table image segmentation unit, used for segmenting the table image to obtain basic grid units corresponding to the table image;

[0041] A feature determination unit, used for determining, for each basic grid unit, a visual feature of the basic grid unit and a text semantic feature of each basic grid unit;

[0042] A structural information determination unit, used to determine structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit;

[0043] The table generating unit is used to generate a table corresponding to the table image according to the structural information of each basic grid unit.

[0044] Optionally, the table image segmentation unit includes:

[0045] A visual feature extraction unit, used for extracting visual features from the table image to obtain visual features of the table image;

[0046] A table line determination unit, configured to determine table line position information in the table image based on visual features of the table image;

[0047] The segmentation unit is used to segment the table image based on the table line position information in the table image to obtain the basic grid unit corresponding to the table image.

[0048] Optionally, the feature determination unit includes a visual feature determination unit, and the visual feature determination unit is specifically configured to:

[0049] Acquire visual features of the table image;

[0050] The visual features of the basic grid unit are extracted from the visual features of the table image.

[0051] Optionally, the feature determination unit includes a text semantic feature determination unit, and the text semantic feature determination unit is specifically used to:

[0052] Identifying text content corresponding to the basic grid unit;

[0053] Semantic features are extracted from the text content to obtain text semantic features of the basic grid unit.

[0054] Optionally, the structure information determining unit includes:

[0055] A global feature determination unit, used to determine the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit;

[0056] The structure information determination subunit is used to determine the structure information of each basic grid unit based on the global characteristics of each basic grid unit.

[0057] Optionally, the global feature determination unit includes:

[0058] An initial global feature determination unit, used for fusing the visual features and text semantic features of each basic grid unit to obtain the initial global features of the basic grid unit;

[0059] An adjustment unit, used for adjusting the initial global features of each basic grid unit based on the mutual dependence between the initial global features of each basic grid unit, so as to obtain the adjusted global features of each basic grid unit;

[0060] The global feature determination subunit is used to use the adjusted global feature of each basic grid unit as the global feature of the basic grid unit.

[0061] Optionally, the structure information determination subunit includes:

[0062] A decoding unit is used to decode each basic grid unit based on the global features of each basic grid unit to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit.

[0063] Optionally, the decoding unit includes:

[0064] A decoding network application unit is used to input the global features of each basic grid unit into a decoding network, and the decoding network decodes each basic grid unit in sequence from left to right and from top to bottom to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, as well as the upper node and left node of the basic grid unit.

[0065] Optionally, the decoding network decodes the basic grid unit as follows:

[0066] Decoding the global features of the basic grid unit to obtain the decoded features of the basic grid unit and the category of the basic grid unit, where the category is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image;

[0067] Get the decoded features of the decoded basic grid cells;

[0068] Using the decoding feature of the basic grid unit as a query and the decoding feature of the decoded basic grid unit as a key, determining the upper decoding feature and the left decoding feature of the basic grid unit from the decoding features of the decoded basic grid unit;

[0069] Based on the upper decoding feature and the left decoding feature of the basic grid unit, an upper node and a left node of the basic grid unit are determined.

[0070] A table image processing device, comprising a memory and a processor;

[0071] The memory is used to store programs;

[0072] The processor is used to execute the program to implement the various steps of the table image processing method as described above.

[0073] A readable storage medium stores a computer program, which, when executed by a processor, implements the various steps of the table image processing method described above.

[0074] By means of the above technical scheme, the present application discloses a table image processing method, related equipment and readable storage medium. After obtaining the table image to be processed, the table image is first segmented to obtain the basic grid units corresponding to the table image, and then the visual features of each basic grid unit and the text semantic features of each basic grid unit are determined. Then, based on the visual features of each basic grid unit and the text semantic features of each basic grid unit, the structural information of each basic grid unit is determined. Finally, according to the structural information of each basic grid unit, the table corresponding to the table image is generated. In this scheme, when converting the table image into a table, the visual information and text semantic information of the table image are considered at the same time, which improves the accuracy of the converted table compared to converting the table using only the visual information of the table image. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0076] Figure 1 A schematic diagram of a flow chart of a table image processing method disclosed in an embodiment of the present application;

[0077] Figure 2 A schematic diagram of table image segmentation disclosed in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of text semantic features of each basic grid unit disclosed in an embodiment of the present application;

[0079] Figure 4 A decoding schematic diagram disclosed in an embodiment of the present application;

[0080] Figure 5 A schematic diagram of the structure of a table image processing device disclosed in an embodiment of the present application;

[0081] Figure 6 This is a hardware structure block diagram of a table image processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] Next, the table image processing method provided by the present application is introduced through the following embodiments.

[0084] Reference Figure 1 , Figure 1 The following is a flow chart of a table image processing method disclosed in an embodiment of the present application. The method may include:

[0085] Step S101: Obtain the table image to be processed.

[0086] In this application, the table image to be processed can be a table image in any scenario (for example, a scene of taking a photo or scanning a picture), and this application does not impose any limitation on this. The style of the table image to be processed can also be any type, such as a wired table image, a semi-wired table image, a wireless table image, etc., and this application does not impose any limitation on this.

[0087] Step S102: segment the table image to obtain basic grid units corresponding to the table image.

[0088] In the present application, segmenting the table image to obtain the basic grid units corresponding to the table image can avoid the problem of reducing the computing speed and increasing the consumption of video memory resources by processing each pixel point of the table image as the basic unit. Therefore, after obtaining the table image to be processed, the table image can also be segmented to obtain the basic grid units corresponding to the table image.

[0089] In the present application, various image segmentation techniques may be used to segment the table image to obtain basic grid units corresponding to the table image, and the present application does not impose any limitation on this.

[0090] Step S103: for each basic grid unit, determining the visual features of the basic grid unit and the text semantic features of each basic grid unit.

[0091] For the table structure, in addition to the dividing lines in the table, the text content in the table is also of great significance. Therefore, in the present application, after segmenting the table image and obtaining the basic grid units corresponding to the table image, the visual features of each basic grid unit and the text semantic features of each basic grid unit can also be determined.

[0092] Step S104: Determine structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit.

[0093] In the present application, after determining the visual features of each basic grid unit and the textual semantic features of each basic grid unit, the structural information of each basic grid unit can be determined based on the visual features of each basic grid unit and the semantic features of each basic grid unit.

[0094] Step S104: generating a table corresponding to the table image according to the structural information of each basic grid unit.

[0095] In the present application, after determining the structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit, the structural information of each basic grid unit can be parsed to determine the entity cells contained in the table corresponding to the table image and the spatial relationship between the entity cells, and based on this, generate the table corresponding to the table image.

[0096] This embodiment discloses a table image processing method. After obtaining a table image to be processed, the table image is first segmented to obtain basic grid units corresponding to the table image. Then, the visual features of each basic grid unit and the text semantic features of each basic grid unit are determined. Then, based on the visual features of each basic grid unit and the text semantic features of each basic grid unit, the structural information of each basic grid unit is determined. Finally, according to the structural information of each basic grid unit, a table corresponding to the table image is generated. In this solution, when converting a table image into a table, both the visual information and the text semantic information of the table image are considered. Compared with converting a table using only the visual information of the table image, the accuracy of the converted table is improved.

[0097] In one embodiment of the present application, a specific implementation method of segmenting the table image in step S102 to obtain basic grid units corresponding to the table image is described, and the method may include the following steps:

[0098] Step S201: extracting visual features of the table image to obtain visual features of the table image.

[0099] In the present application, a feature extraction network may be used to extract visual features of the table image to obtain visual features of the table image. As an implementable method, ResNet-34 and FPN may be used as backbones to extract visual features of the table image to obtain visual features of the table image.

[0100] Step S202: Determine table line position information in the table image based on the visual features of the table image.

[0101] In the present application, the table line position information in the table image includes row line position information and column line position information. In the present application, a table line detection network can be used to perform table line detection on the visual features of the table image to determine the table line position information in the table image.

[0102] Step S203: segmenting the table image based on the table line position information in the table image to obtain basic grid units corresponding to the table image.

[0103] In the present application, after obtaining the table line position information in the table image, the table image can be segmented based on the table line position information in the table image to obtain the basic grid units corresponding to the table image.

[0104] For easier understanding, please refer to the attached Figure 2 , Figure 2 Schematic diagram of a table image segmentation disclosed in the embodiment of the present application. Figure 2 As shown, the feature extraction network performs visual feature extraction on the table image to be processed to obtain the visual feature P2 of the table image, and the table line detection network performs table line detection on the visual feature P2 of the table image to determine the row table lines and column grid lines in the table image, and obtains the segmented table image based on the row table lines and column grid lines in the table image, wherein each small square in the segmented table image is a basic grid unit.

[0105] In one embodiment of the present application, the method of determining the visual features of the basic grid unit in step S103 is described, and the method may include the following steps:

[0106] Step S301: Acquire the visual features of the table image.

[0107] In the present application, the visual features of the table image may be the visual features of the table image obtained by extracting the visual features of the table image in the above step S201.

[0108] Step S302: extracting visual features of the basic grid unit from the visual features of the table image.

[0109] In the present application, as an implementable method, the visual features of the basic grid unit can be extracted from the visual features of the table image through RoIAlign.

[0110] In one embodiment of the present application, the method of determining the text semantic features of the basic grid unit in step S103 is described, and the method may include the following steps:

[0111] Step S401: Identify the text content corresponding to the basic grid unit.

[0112] In the present application, a recognizer may be used to recognize text content corresponding to the basic grid unit. The recognizer may be any realizable recognizer, and the present application does not impose any limitation on this.

[0113] Step S402: extracting semantic features from the text content to obtain text semantic features of the basic grid unit.

[0114] In the present application, a semantic feature extraction network can be used to extract semantic features from the text content to obtain text semantic features of the basic grid unit. As an implementable method, in the present application, a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model can be used to extract semantic features from the text content to obtain text semantic features of the basic grid unit.

[0115] For easier understanding, please refer to the attached Figure 3 , Figure 3 Schematic diagram of text semantic features of each basic grid unit disclosed in the embodiment of this application. Figure 3 As shown, by identifying each basic grid unit through an identifier, the text content corresponding to each basic grid unit can be obtained, and by extracting semantic features of the text content corresponding to each basic grid unit, the text semantic features of each basic grid unit can be obtained.

[0116] In one embodiment of the present application, the implementation method of determining the structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit in step S104 is described, and the method may include the following steps:

[0117] Step S501: Determine the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit.

[0118] As an implementable method, determining the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit includes:

[0119] Step S5011: For each basic grid unit, the visual features and text semantic features of the basic grid unit are fused to obtain the initial global features of the basic grid unit.

[0120] In this step, the visual features and text semantic features of each basic grid unit can be spliced ​​to obtain the initial global features of the basic grid unit. Of course, in this step, the visual features and text semantic features of each basic grid unit can also be fused in other forms to obtain the initial global features of the basic grid unit, and this application does not impose any limitation on this.

[0121] Step S5012: Based on the mutual dependence between the initial global features of each basic grid unit, the initial global features of each basic grid unit are adjusted to obtain the adjusted global features of each basic grid unit.

[0122] In the present application, as an implementable method, the self-attention mechanism of the Transformer network can be used to determine the mutual dependence between the initial global features of each basic grid unit, and the initial global features of each basic grid unit can be adjusted to obtain the adjusted global features of each basic grid unit. Of course, in this step, other methods can also be used to adjust the initial global features of each basic grid unit based on the mutual dependence between the initial global features of each basic grid unit, and this application does not impose any limitation on this.

[0123] Step S5013: using the adjusted global features of each basic grid unit as the global features of the basic grid unit.

[0124] In this step, the global features of each basic grid unit obtained can reflect the interdependence between the basic grid units, so that the global features of each basic grid unit have a global receptive field.

[0125] Step S502: Determine the structural information of each basic grid unit based on the global features of each basic grid unit.

[0126] As an implementable method, the determining the structural information of each basic grid unit based on the global features of each basic grid unit includes:

[0127] Based on the global features of each basic grid unit, each basic grid unit is decoded to determine the structural information of each basic grid unit. The structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit.

[0128] Since there are merged cells and split cells in the table, some of the basic grid cells determined after the aforementioned segmentation of the table image are not physical cells. In view of this, in this application, the category of the basic grid cell can be preset, and the category is used to characterize whether the basic grid cell is a physical cell in the table corresponding to the table image. For example, if the basic grid cell category is a physical cell, it indicates that the basic grid cell is a physical cell in the table corresponding to the table image, and if the basic grid cell category is a line break virtual body, it indicates that the basic grid cell is not a physical cell in the table corresponding to the table image.

[0129] Based on the above settings, in this application, the structural information of the basic grid cell can be represented by whether the basic grid cell is a physical cell in the table corresponding to the table image. In addition, the table is a highly structured entity, and each of its internal cells has a corresponding adjacent node. Figure 2 In the table image shown, the upper node of the cell with the content of 'Simple' is 'TEDS' and the left node is 'System'; the upper node of the cell with the content of 'Complex' is 'TEDS' and the left node is 'Simple'. Therefore, in this application, the adjacent nodes of the basic grid unit can also be used to characterize the structural information of the basic grid unit.

[0130] It should be noted that the adjacent node can be any one or more of the upper node, the lower node, the left node, and the right node. However, people are accustomed to reading tables from left to right and from top to bottom. In order to match the reading habit, in this application, the adjacent node can be the upper node and the left node.

[0131] As an implementable method, decoding each basic grid unit based on the global features of each basic grid unit to determine the structural information of each basic grid unit includes:

[0132] The global features of each basic grid unit are input into a decoding network, and the decoding network decodes each basic grid unit in order from left to right and from top to bottom to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the upper node and left node of the basic grid unit. In the present application, the decoding network is a tree decoding network.

[0133] For each basic grid unit, the decoding method of the basic grid unit by the decoding network may include the following steps:

[0134] Step S601: Decode the global features of the basic grid unit to obtain the decoded features of the basic grid unit and the category of the basic grid unit, where the category is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image.

[0135] In this application, the GRU (Gated Recurrent Unit) structure can be used inside the decoding network. Based on this, it is assumed that the decoding feature of the current basic grid unit is h t , then it is necessary to decode the feature h based on the previous basic grid unit t-1 And the reference vector c when decoding the previous basic grid unit t-1 Estimate the hidden state of the current base grid cell Then, As query, the global features of the basic grid unit are used as key and value to obtain the reference vector c that should be paid attention to at the current basic grid unit decoding time. t : Using c t and The decoding feature h of the current basic grid unit can be updated t :

[0136] Step S602: Obtain the decoded features of the decoded basic grid units.

[0137] In the present application, a stack (Memory) may be preset in the decoding network to store the decoding features of the decoded basic grid units.

[0138] Step S603: using the decoding feature of the basic grid unit as a query and the decoding feature of the decoded basic grid unit as a key, determining the upper decoding feature and the left decoding feature of the basic grid unit from the decoding features of the decoded basic grid unit.

[0139] For easier understanding, please refer to the attached Figure 4, Figure 4 Schematic diagram of decoding disclosed in the embodiment of this application. Figure 4 As shown, the basic grid cell with the content 'Simple' has a decoding feature of h3, the decoding features of the decoded basic grid cell are h0h1h2, its left decoding feature is h0, and its upper decoding feature is h1.

[0140] Step S604: Determine the upper node and the left node of the basic grid unit based on the upper decoding feature and the left decoding feature of the basic grid unit.

[0141] For ease of understanding, in step S603, the basic grid unit with the content of 'Simple' is taken as an example, its left decoding feature is h0, h0 is the decoding feature of the basic grid unit with the content of 'System', and its left node is the basic grid unit with the content of 'System'. Its upper decoding feature is h1. h1 is the decoding feature of the basic grid unit with the content of 'TEDS', and its upper node is the basic grid unit with the content of 'TEDS'.

[0142] The table image processing device disclosed in the embodiment of the present application is described below. The table image processing device described below and the table image processing method described above can be referenced to each other.

[0143] Reference Figure 5 , Figure 5 FIG. 1 is a schematic diagram of the structure of a table image processing device disclosed in an embodiment of the present application. Figure 5 As shown, the table image processing device may include:

[0144] An acquisition unit 51 is used to acquire a table image to be processed;

[0145] A table image segmentation unit 52, used to segment the table image to obtain basic grid units corresponding to the table image;

[0146] A feature determination unit 53, configured to determine, for each basic grid unit, a visual feature of the basic grid unit and a text semantic feature of each basic grid unit;

[0147] A structure information determining unit 54, configured to determine the structure information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit;

[0148] The table generating unit 55 is used to generate a table corresponding to the table image according to the structural information of each basic grid unit.

[0149] As an implementable embodiment, the table image segmentation unit includes:

[0150] A visual feature extraction unit, used for extracting visual features from the table image to obtain visual features of the table image;

[0151] A table line determination unit, configured to determine table line position information in the table image based on visual features of the table image;

[0152] The segmentation unit is used to segment the table image based on the table line position information in the table image to obtain the basic grid unit corresponding to the table image.

[0153] As an implementable embodiment, the feature determination unit includes a visual feature determination unit, and the visual feature determination unit is specifically used to:

[0154] Acquire visual features of the table image;

[0155] The visual features of the basic grid unit are extracted from the visual features of the table image.

[0156] As an implementable method, the feature determination unit includes a text semantic feature determination unit, and the text semantic feature determination unit is specifically used to:

[0157] Identifying text content corresponding to the basic grid unit;

[0158] Semantic features are extracted from the text content to obtain text semantic features of the basic grid unit.

[0159] As an implementable embodiment, the structure information determining unit includes:

[0160] A global feature determination unit, used to determine the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit;

[0161] The structure information determination subunit is used to determine the structure information of each basic grid unit based on the global characteristics of each basic grid unit.

[0162] As an implementable embodiment, the global feature determination unit includes:

[0163] An initial global feature determination unit, used for fusing the visual features and text semantic features of each basic grid unit to obtain the initial global features of the basic grid unit;

[0164] An adjustment unit, used for adjusting the initial global features of each basic grid unit based on the mutual dependence between the initial global features of each basic grid unit, so as to obtain the adjusted global features of each basic grid unit;

[0165] The global feature determination subunit is used to use the adjusted global feature of each basic grid unit as the global feature of the basic grid unit.

[0166] As an implementable embodiment, the structure information determination subunit includes:

[0167] A decoding unit is used to decode each basic grid unit based on the global features of each basic grid unit to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit.

[0168] As an implementable embodiment, the decoding unit includes:

[0169] A decoding network application unit is used to input the global features of each basic grid unit into a decoding network, and the decoding network decodes each basic grid unit in sequence from left to right and from top to bottom to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, as well as the upper node and left node of the basic grid unit.

[0170] As an implementable method, the decoding network decodes the basic grid unit as follows:

[0171] Decoding the global features of the basic grid unit to obtain the decoded features of the basic grid unit and the category of the basic grid unit, where the category is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image;

[0172] Get the decoded features of the decoded basic grid cells;

[0173] Using the decoding feature of the basic grid unit as a query and the decoding feature of the decoded basic grid unit as a key, determining the upper decoding feature and the left decoding feature of the basic grid unit from the decoding features of the decoded basic grid unit;

[0174] Based on the upper decoding feature and the left decoding feature of the basic grid unit, an upper node and a left node of the basic grid unit are determined.

[0175] Reference Figure 6 , Figure 6 The hardware structure diagram of the table image processing device provided in the embodiment of the present application is shown in FIG. Figure 6,The hardware structure of the table image processing device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0176] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0177] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0178] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0179] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0180] Get the table image to be processed;

[0181] Segmenting the table image to obtain basic grid units corresponding to the table image;

[0182] For each basic grid unit, determining a visual feature of the basic grid unit and a text semantic feature of each basic grid unit;

[0183] Determine structural information of each basic grid unit based on visual features of each basic grid unit and text semantic features of each basic grid unit;

[0184] A table corresponding to the table image is generated according to the structural information of each basic grid unit.

[0185] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0186] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0187] Get the table image to be processed;

[0188] Segmenting the table image to obtain basic grid units corresponding to the table image;

[0189] For each basic grid unit, determining a visual feature of the basic grid unit and a text semantic feature of each basic grid unit;

[0190] Determine structural information of each basic grid unit based on visual features of each basic grid unit and text semantic features of each basic grid unit;

[0191] A table corresponding to the table image is generated according to the structural information of each basic grid unit.

[0192] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0194] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0195] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A table image processing method, characterized in that: The method comprises: Get the table image to be processed; Segmenting the table image to obtain basic grid units corresponding to the table image; For each basic grid unit, determining a visual feature of the basic grid unit and a text semantic feature of each basic grid unit; Determine the structural information of each basic grid unit based on the visual features of each basic grid unit and the textual semantic features of each basic grid unit; the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit; A table corresponding to the table image is generated according to the structural information of each basic grid unit.

2. The method according to claim 1, characterized in that The step of segmenting the table image to obtain basic grid units corresponding to the table image includes: Extracting visual features of the table image to obtain visual features of the table image; Determining table line position information in the table image based on visual features of the table image; Based on the table line position information in the table image, the table image is segmented to obtain basic grid units corresponding to the table image.

3. The method according to claim 1, characterized in that Determining the visual features of the basic grid unit includes: Acquire visual features of the table image; The visual features of the basic grid unit are extracted from the visual features of the table image.

4. The method according to claim 1, characterized in that Determining the text semantic features of the basic grid unit includes: Identifying text content corresponding to the basic grid unit; Semantic features are extracted from the text content to obtain text semantic features of the basic grid unit.

5. The method according to claim 1, characterized in that The determining of the structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit includes: Determine the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit; Based on the global characteristics of each basic grid unit, the structural information of each basic grid unit is determined.

6. The method according to claim 5, characterized in that The determining of the global features of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit includes: For each basic grid unit, the visual features and text semantic features of the basic grid unit are fused to obtain the initial global features of the basic grid unit; Based on the mutual dependence between the initial global features of each basic grid unit, the initial global features of each basic grid unit are adjusted to obtain the adjusted global features of each basic grid unit; The adjusted global features of each basic grid unit are used as the global features of the basic grid unit.

7. The method according to claim 5, characterized in that The determining of the structural information of each basic grid unit based on the global features of each basic grid unit includes: Based on the global features of each basic grid unit, each basic grid unit is decoded to determine the structural information of each basic grid unit. The structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit.

8. The method according to claim 7, characterized in that The decoding of each basic grid unit based on the global features of each basic grid unit to determine the structural information of each basic grid unit includes: The global features of each basic grid unit are input into a decoding network, and the decoding network decodes each basic grid unit in sequence from left to right and from top to bottom to determine the structural information of each basic grid unit, and the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, as well as the upper node and left node of the basic grid unit.

9. The method according to claim 8, characterized in that For each basic grid unit, the decoding network decodes the basic grid unit as follows: Decoding the global features of the basic grid unit to obtain the decoded features of the basic grid unit and the category of the basic grid unit, where the category is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image; Get the decoded features of the decoded basic grid cells; Using the decoding feature of the basic grid unit as a query and the decoding feature of the decoded basic grid unit as a key, determining the upper decoding feature and the left decoding feature of the basic grid unit from the decoding features of the decoded basic grid unit; Based on the upper decoding feature and the left decoding feature of the basic grid unit, an upper node and a left node of the basic grid unit are determined.

10. A table image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring a table image to be processed; A table image segmentation unit, used for segmenting the table image to obtain basic grid units corresponding to the table image; A feature determination unit, used for determining, for each basic grid unit, a visual feature of the basic grid unit and a text semantic feature of each basic grid unit; A structural information determination unit, used to determine the structural information of each basic grid unit based on the visual features of each basic grid unit and the text semantic features of each basic grid unit; the structural information of each basic grid unit is used to indicate whether the basic grid unit is a physical cell in the table corresponding to the table image, and the adjacent nodes of the basic grid unit; The table generating unit is used to generate a table corresponding to the table image according to the structural information of each basic grid unit.

11. A table image processing device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the table image processing method as described in any one of claims 1 to 9.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the table image processing method according to any one of claims 1 to 9 is implemented.

Images