Table recognition methods, devices, electronic equipment, media and program products

Abstract

This specification discloses a table recognition method, apparatus, electronic device, medium, and program product. The method includes: acquiring a target image containing a table; determining global relationship information corresponding to the table based on the target image, the global relationship information including relationships between target texts and target cells in the table; determining local relationship information corresponding to the table based on the global relationship information, the local relationship information including relationships between target texts in first regions and target cells in second regions of the table; and finally reconstructing the table in the target image based on the local relationship information.

Description

Technical Field

[0001] This specification relates to the field of information processing technology, and in particular to a table recognition method, apparatus, electronic device, medium, and program product. Background Technology

[0002] Across various industries, there is a significant amount of work involved in extracting and inputting forms. For example, in the insurance industry, claims processing requires extracting detailed invoice information and related data. Relying entirely on manual extraction is not only costly but also prone to errors. Currently, in the actual process of machine invoice form recognition, invoice forms come in various formats. Rule-based invoice form recognition methods or traditional image recognition methods require continuous adjustment of extraction rules or thresholds set by traditional image algorithms to meet the needs of recognizing new invoice form types. Furthermore, using deep learning technology to detect lines on invoice forms and combining it with text recognition technology can only solve the recognition and input of wired invoice forms, and cannot solve the recognition and reconstruction of wireless forms. Summary of the Invention

[0003] This specification provides a table recognition method, apparatus, electronic device, medium, and program product. By combining global and local relationship information within the table to reconstruct the table, the accuracy and robustness of table recognition can be improved. The above technical solution is as follows:

[0004] Firstly, embodiments of this specification provide a table recognition method, including:

[0005] Get the target image containing the table;

[0006] Based on the target image above, determine the global relationship information corresponding to the table above; the global relationship information includes the relationship between each target text in the table above and the relationship between each target cell;

[0007] Based on the global relationship information mentioned above, the local relationship information corresponding to the table above is determined; the local relationship information includes the relationship between each target text in each first area of ​​the table above and the relationship between each target cell in each second area of ​​the table above.

[0008] The table in the target image above is reconstructed based on the aforementioned local relationship information.

[0009] In one possible implementation, determining the global relationship information corresponding to the table based on the target image includes:

[0010] Based on the target text in the target image, the target location corresponding to the target text, and the target image, determine the multimodal information corresponding to each target text in the table above;

[0011] Based on the aforementioned multimodal information, the global relationship information corresponding to the above table is determined.

[0012] In one possible implementation, the aforementioned multimodal information includes text modal information, location modal information, and image feature modal information corresponding to the target text.

[0013] The above-mentioned determination of the multimodal information corresponding to each target character in the above table based on the target text in the above target image, the target position corresponding to the target text, and the above target image includes:

[0014] Based on the target text in the target image and the target position corresponding to the target text, extract the target image features corresponding to the target text from the target image.

[0015] The text modality information is determined based on the target text in the target image, the position modality information is determined based on the target position corresponding to the target text, and the image feature modality information is determined based on the target image features corresponding to the target text.

[0016] In one possible implementation, determining the global relationship information corresponding to the above table based on the above multimodal information includes:

[0017] The multimodal information corresponding to each target character in the above table is input into the encoder, and the target feature information corresponding to the above table is output. The target feature information includes the first target multimodal feature information of the multimodal information corresponding to each target character in the above table and the global relationship information corresponding to the above table.

[0018] In one possible implementation, the aforementioned local relationship information includes the target positional relationship between target texts in each first region of the table and the target row and column relationship between target cells in each second region of the table.

[0019] The above determination of the local relationship information corresponding to the above table based on the above global relationship information includes:

[0020] The target positional relationship is determined based on the above target feature information; the above target positional relationship is used to characterize whether each target text in each first area of ​​the above table is located in the same cell;

[0021] The row and column relationships of the targets are determined based on the above-mentioned target positional relationships and the above-mentioned first target multimodal feature information.

[0022] In one possible implementation, determining the target positional relationship based on the aforementioned target feature information includes:

[0023] Based on the above target feature information, a target single-character sub-graph set is constructed; the above target single-character sub-graph set includes at least one target single-character sub-graph; the above target single-character sub-graph includes multiple target text nodes located in the same area of ​​the above table and the connection relationship between the multiple target text nodes; the above target text node includes the first target multimodal feature information corresponding to the above target text;

[0024] The above target single-character sub-image set is input into the first graph convolutional neural network, which outputs the target positional relationship between each target character node in the above target single-character sub-image; the above first graph convolutional neural network is trained based on multiple single-character sub-images with known positional relationships between each character node.

[0025] In one possible implementation, determining the target row-column relationship based on the target positional relationship and the first target multimodal feature information includes:

[0026] Based on the aforementioned target positional relationships and the aforementioned first target multimodal feature information, a target cell sub-graphet is constructed; the aforementioned target cell sub-graphet includes at least one target cell sub-graph; the aforementioned target cell sub-graph includes multiple target cell nodes located in the same area of ​​the aforementioned table and the connection relationships between the multiple target cell nodes; the aforementioned target cell node includes the second target multimodal feature information corresponding to the aforementioned target cell; the aforementioned second target multimodal feature information is obtained based on the first target multimodal feature information corresponding to all target text located in the aforementioned target cell node;

[0027] The above target cell subgraph set is input into the second graph convolutional neural network, which outputs the target row and column relationships between each target cell node in the above target cell subgraph. The above second graph convolutional neural network is trained based on multiple cell subgraphs with known row and column relationships between each cell node.

[0028] In one possible implementation, the construction of the target cell sub-map set based on the aforementioned target positional relationships and the aforementioned first target multimodal feature information includes:

[0029] Based on the above target positional relationships, the target cell information corresponding to each target cell in the above table is determined; the target cell information includes the target text located in the target cell.

[0030] By fusing the first target multimodal feature information corresponding to each target text in the above target cell, the above second target multimodal feature information is obtained;

[0031] Based on the aforementioned second target multimodal feature information, a target cell sub-map set is constructed.

[0032] In one possible implementation, the plurality of target text nodes includes a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node; the target positional relationship is used to characterize the positional relationship between the central target text node and the first and second target text nodes.

[0033] In one possible implementation, the plurality of target cell nodes include a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node; the target row-column relationship is used to characterize the row-column relationship between the central target cell node and the first and second target cell nodes.

[0034] In one possible implementation, the above-mentioned reconstruction of the table in the target image based on the above-mentioned local relationship information includes: reconstructing the table in the target image according to the above-mentioned target positional relationship and the above-mentioned target row and column relationship in the table.

[0035] In one possible implementation, the reconstruction of the table in the target image based on the aforementioned local relational information includes:

[0036] Based on the aforementioned target row and column relationships and the aforementioned target cell sub-atlas, a target row atlas and a target column atlas are generated; the aforementioned target row atlas includes the target row atlas corresponding to each row in the aforementioned table; the aforementioned target row atlas includes the target cell node corresponding to the target cell located in the same row; the aforementioned target column atlas includes the target column atlas corresponding to each column in the aforementioned table; the aforementioned target column atlas includes the target cell node corresponding to the target cell located in the same column;

[0037] Based on the above target row atlas, the above target column atlas, and the target cell information corresponding to the above target cell nodes, the table in the above target image is reconstructed.

[0038] Secondly, embodiments of this specification provide a table recognition device, including:

[0039] The acquisition module is used to acquire the target image containing the table;

[0040] The first determining module is used to determine the global relationship information corresponding to the table based on the target image; the global relationship information includes the relationship between each target text in the table and the relationship between each target cell.

[0041] The second determining module is used to determine the local relationship information corresponding to the table based on the global relationship information mentioned above; the local relationship information includes the relationship between each target text in each first area of ​​the table and the relationship between each target cell in each second area of ​​the table.

[0042] The reconstruction module is used to reconstruct the table in the target image based on the aforementioned local relationship information.

[0043] In one possible implementation, the first determining module mentioned above includes:

[0044] The first determining unit is used to determine the multimodal information corresponding to each target character in the table based on the target characters in the target image, the target positions corresponding to the target characters, and the target image.

[0045] The second determining unit is used to determine the global relationship information corresponding to the above table based on the above multimodal information.

[0046] In one possible implementation, the aforementioned multimodal information includes text modal information, location modal information, and image feature modal information corresponding to the target text.

[0047] The aforementioned first determining unit includes:

[0048] An extraction subunit is used to extract the target image features corresponding to the target text from the target image based on the target text in the target image and the target position corresponding to the target text.

[0049] The determination subunit is used to determine the text modality information based on the target text in the target image, to determine the position modality information based on the target position corresponding to the target text, and to determine the image feature modality information based on the target image features corresponding to the target text.

[0050] In one possible implementation, the second determining unit is specifically used for:

[0051] The multimodal information corresponding to each target character in the above table is input into the encoder, and the target feature information corresponding to the above table is output. The target feature information includes the first target multimodal feature information of the multimodal information corresponding to each target character in the above table and the global relationship information corresponding to the above table.

[0052] In one possible implementation, the aforementioned local relationship information includes the target positional relationship between target texts in each first region of the table and the target row and column relationship between target cells in each second region of the table.

[0053] The second determining module mentioned above includes:

[0054] The third determining unit is used to determine the target positional relationship based on the target feature information mentioned above; the target positional relationship is used to characterize whether each target text in each first area of ​​the table is located in the same cell.

[0055] The fourth determining unit is used to determine the row and column relationship of the target based on the above target position relationship and the above target multimodal feature information.

[0056] In one possible implementation, the third determining unit mentioned above includes:

[0057] The first construction subunit is used to construct a target single-character sub-graph set based on the aforementioned target feature information; the aforementioned target single-character sub-graph set includes at least one target single-character sub-graph; the aforementioned target single-character sub-graph includes multiple target text nodes located in the same area of ​​the aforementioned table and the connection relationship between the multiple target text nodes; the aforementioned target text node includes the first target multimodal feature information corresponding to the aforementioned target text;

[0058] The first convolutional sub-unit is used to input the above target single-character sub-graph set into the first graph convolutional neural network and output the target positional relationship between each target character node in the above target single-character sub-graph; the above first graph convolutional neural network is trained based on multiple single-character sub-graphs with known positional relationships between each character node.

[0059] In one possible implementation, the fourth determining unit mentioned above includes:

[0060] The second construction subunit is used to construct a target cell sub-graphet based on the aforementioned target positional relationships and the aforementioned first target multimodal feature information; the aforementioned target cell sub-graphet includes at least one target cell sub-graph; the aforementioned target cell sub-graph includes multiple target cell nodes located in the same area of ​​the aforementioned table and the connection relationships between the multiple target cell nodes; the aforementioned target cell node includes the second target multimodal feature information corresponding to the aforementioned target cell; the aforementioned second target multimodal feature information is obtained based on the first target multimodal feature information corresponding to all target text located in the aforementioned target cell node;

[0061] The second convolutional subunit is used to input the above target cell subgraph set into the second graph convolutional neural network and output the target row and column relationship between each target cell node in the above target cell subgraph; the above second graph convolutional neural network is trained based on multiple cell subgraphs with known row and column relationships between each cell node.

[0062] In one possible implementation, the second building block described above is specifically used for:

[0063] Based on the above target positional relationships, the target cell information corresponding to each target cell in the above table is determined; the target cell information includes the target text located in the target cell.

[0064] By fusing the first target multimodal feature information corresponding to each target text in the above target cell, the above second target multimodal feature information is obtained;

[0065] Based on the aforementioned second target multimodal feature information, a target cell sub-map set is constructed.

[0066] In one possible implementation, the plurality of target text nodes includes a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node; the target positional relationship is used to characterize the positional relationship between the central target text node and the first and second target text nodes.

[0067] In one possible implementation, the plurality of target cell nodes include a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node; the target row-column relationship is used to characterize the row-column relationship between the central target cell node and the first and second target cell nodes.

[0068] In one possible implementation, the above-mentioned refactoring module is specifically used for:

[0069] Reconstruct the table in the target image based on the target positional relationships and target row-column relationships in the table above.

[0070] In one possible implementation, the above-mentioned refactoring module includes:

[0071] The generation unit is used to generate a target row map set and a target column map set based on the target row-column relationship and the target cell sub-map set. The target row map set includes the target row map corresponding to each row in the table. The target row map includes the target cell node corresponding to the target cell in the same row. The target column map set includes the target column map corresponding to each column in the table. The target column map includes the target cell node corresponding to the target cell in the same column.

[0072] The reconstruction unit is used to reconstruct the table in the target image based on the target row atlas, the target column atlas, and the target cell information corresponding to the target cell node.

[0073] Thirdly, embodiments of this specification provide an electronic device, including: a processor and a memory;

[0074] The processor is connected to the memory.

[0075] The aforementioned memory is used to store executable program code;

[0076] The processor reads the executable program code stored in the memory to run the program corresponding to the executable program code, so as to execute the method provided by the first aspect of the embodiments of this specification or any possible implementation of the first aspect.

[0077] Fourthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method provided by the first aspect of the embodiments of this specification or any possible implementation thereof.

[0078] Fifthly, embodiments of this specification provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to execute the table recognition method provided by the first aspect of the embodiments of this specification or any possible implementation thereof.

[0079] This embodiment of the specification acquires a target image containing a table, and determines the global relationship information corresponding to the table based on the target image. The global relationship information includes the relationship between each target text in the table and the relationship between each target cell. Then, based on the global relationship information, it determines the local relationship information corresponding to the table. The local relationship information includes the relationship between each target text in each first region of the table and the relationship between each target cell in each second region of the table. Finally, based on the local relationship information, it reconstructs the table in the target image. By fully mining the global and local relationship information between target texts and between target cells in the table, the global and local relationship information in the table are combined to reconstruct the table. This enables the recognition and reconstruction of both wired and wireless tables, improving the accuracy and robustness of table recognition. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1A schematic diagram of the architecture of a table recognition system provided for an exemplary embodiment of this specification;

[0082] Figure 2 A flowchart illustrating a table recognition method provided for an exemplary embodiment of this specification;

[0083] Figure 3 A flowchart illustrating another table recognition method provided as an exemplary embodiment of this specification;

[0084] Figure 4 A schematic diagram illustrating the implementation process of determining multimodal information corresponding to a target text, provided as an exemplary embodiment of this specification;

[0085] Figure 5 A flowchart illustrating a method for determining local relation information corresponding to a table, provided as an exemplary embodiment of this specification;

[0086] Figure 6 This specification provides a flowchart illustrating a method for determining target positional relationships as an exemplary embodiment.

[0087] Figure 7 A schematic diagram of a target single-word subgraph provided for an exemplary embodiment of this specification;

[0088] Figure 8 This specification provides a flowchart illustrating a method for determining target row-column relationships as an exemplary embodiment.

[0089] Figure 9 A schematic diagram illustrating a process for constructing a target cell sub-atlas, provided as an exemplary embodiment of this specification;

[0090] Figure 10 A schematic diagram illustrating the implementation process of a table recognition method provided for an exemplary embodiment of this specification;

[0091] Figure 11 A schematic diagram of the structure of a form recognition device provided for an exemplary embodiment of this specification;

[0092] Figure 12 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0093] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0094] The terms "first," "second," "third," etc., used in this specification, claims, and the foregoing drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0095] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, target images containing tables involved in this specification were obtained with full authorization.

[0096] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the architecture of a table recognition system provided for an exemplary embodiment of this specification. Figure 1 As shown, the form recognition system may include: a user terminal 110 and a server 120. Wherein:

[0097] Terminal 110 can be a user terminal, specifically including one or more user terminals. User-version software can be installed on terminal 110 to enable online input of target images, including tables, by the user. Terminal 110 can establish a data relationship with the network and, through the network, establish a data connection with server 120, such as sending target images including tables, receiving target images, and reconstructing tables from them. Terminal 110 can be, but is not limited to, devices such as mobile phones, tablets, and laptops with user-version software installed. It should be noted that any user terminal in the embodiments of this specification can be equipped with a wide-angle camera. This wide-angle camera can be used to acquire target images including tables. The user terminal can crop the target image and send the processed target image to other terminals or server 120.

[0098] Server 120 can be a server capable of providing various table recognition and reconstruction services. It can receive target images, including tables, sent from the network or a user terminal. Based on the target image, it determines the global relationship information corresponding to the table, including the relationships between target text and target cells within the table. Then, based on the global relationship information, it determines the local relationship information corresponding to the table, including the relationships between target text within first regions and target cells within second regions within the table. Finally, based on the local relationship information, it reconstructs the table in the target image. Server 120 can also output the reconstructed table from the target image to a terminal 110, etc. Server 120 can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0099] The recognition and reconstruction of the table in the target image is not limited to the server 120, but can also be performed by the terminal 110. The terminal 110 can acquire a target image containing a table through a wide-angle camera installed on it, or receive a target image containing a table sent by other user terminals via a network, and determine the global relationship information corresponding to the table based on the target image, as well as the local relationship information corresponding to the table based on the global relationship information. Finally, it reconstructs the table in the target image based on the local relationship information. This specification does not specifically limit this embodiment; all subsequent embodiments use the example of the terminal 110 performing the recognition and reconstruction of the table in the target image.

[0100] The network can be a medium that provides a communication link between server 120 and terminal 110, or it can be the Internet, which includes network devices and transmission media, and is not limited thereto. The transmission media can be a wired link, such as, but not limited to, coaxial cable, fiber optic cable, and digital subscriber line (DSL), or a wireless link, such as, but not limited to, wireless fidelity (WIFI), Bluetooth, and mobile device networks.

[0101] Understandably, Figure 1 The number of terminals 110 and servers 120 in the table recognition and reconstruction system shown is merely an example. In a specific implementation, the table recognition and reconstruction system can contain any number of terminals 110 and servers. This specification does not specifically limit this. For example, but not limited to, terminal 110 can be a terminal cluster composed of multiple terminals, and server 120 can be a server cluster composed of multiple servers.

[0102] Next, combine Figure 1 This document describes the table recognition method provided in the embodiments of this specification. Please refer to [link / reference] for details. Figure 2 This is a flowchart illustrating a table recognition method provided in an exemplary embodiment of this specification. Figure 2 As shown, the table recognition method includes the following steps:

[0103] S202, Obtain the target image containing the table.

[0104] Specifically, when a user wants to identify and reconstruct a wired or wireless ticket form, the wide-angle camera installed on the terminal 110 can capture the target image corresponding to the form to be identified. The number of forms in the target image can be one or more, and this embodiment does not limit this.

[0105] Alternatively, in addition to acquiring the target image through the wide-angle camera installed on the terminal 110 for table recognition, the target image stored on the terminal 110 can be directly acquired or the target image can be acquired from the network for table recognition. This specification does not limit the embodiments in this way.

[0106] S204, Determine the global relationship information corresponding to the table based on the target image.

[0107] Specifically, the aforementioned global relational information includes the relationships between target characters in the table to be identified and the relationships between target cells. After acquiring the target image, text recognition can be performed on the target image to obtain the target characters in the table to be identified and the target positions corresponding to each target character. Then, based on the target positions corresponding to each target character in the table, the target image features corresponding to the target positions of the target characters are extracted from the target image. The target characters, their corresponding target positions, and the target image features in the table are encoded into corresponding target character vectors, target position vectors, and target image feature vectors, respectively, thereby obtaining the multimodal information corresponding to each target character in the table. That is, the multimodal information includes the target character vector, target position vector, and target image feature vector corresponding to the target characters. Finally, the relationships between the target characters can be determined directly based on the multimodal information corresponding to each target character in the table, and the relationships between the target cells in the table can be further determined based on the relationships between the target characters in the table, thereby obtaining the global relational information corresponding to the table to be identified.

[0108] Understandably, the aforementioned multimodal information may also include only one or two of the text modal information (target text vector), position modal information (target position vector), and image feature modal information (target image feature vector) corresponding to the target text. This specification does not limit this.

[0109] Optionally, after obtaining the target image containing the table, in order to enhance the quality of the target image and improve the table recognition effect, the target image can be preprocessed first, and then the global relationship corresponding to the table can be determined based on the preprocessed target image. The above preprocessing may include, but is not limited to, image deblurring, image brightness enhancement, image contrast enhancement, image super-resolution reconstruction, and image correction.

[0110] S206, Determine the local relationship information corresponding to the table based on the global relationship information.

[0111] Specifically, the aforementioned local relationship information includes the relationships between target text within each first region of the table and the relationships between target cells within each second region of the table. Both the first and second regions are regions within the table; they can be the same region or different regions, and this specification does not limit this. Each first region includes at least two target texts, and each second region includes at least two target cells. After determining the relationships between the target texts and target cells in the table, the relationships between the target texts within each first region of the table can be determined based on the relationships between the target texts in the table, and the relationships between the target cells within each second region of the table can be determined based on the relationships between the target cells in the table. That is, the local relationship information corresponding to the table can be determined based on the global relationship information in the table. The relationships between the target texts within each first region may include, but are not limited to, the positional relationships between the target texts within the first region, such as, but not limited to, whether the target texts within the first region are located within the same target cell. The relationships between the target cells within each second region may include, but are not limited to, the row and column relationships between the target cells within the second region, such as, but not limited to, whether the target cells within the second region are located in the same row or column.

[0112] S208, Reconstruct the table in the target image based on local relational information.

[0113] Specifically, after determining the local relationship information corresponding to the table to be identified, the table to be identified can be reconstructed based on the local relationship information. That is, the table can be reconstructed based on the positional relationship between each target text in each first area of ​​the table and the row and column relationship between each target cell in each second area.

[0114] This embodiment of the specification acquires a target image containing a table, and determines the global relationship information corresponding to the table based on the target image. The global relationship information includes the relationship between each target text in the table and the relationship between each target cell. Then, based on the global relationship information, it determines the local relationship information corresponding to the table. The local relationship information includes the relationship between each target text in each first region of the table and the relationship between each target cell in each second region of the table. Finally, based on the local relationship information, it reconstructs the table in the target image. By fully mining the global and local relationship information between target texts and between target cells in the table, the global and local relationship information in the table are combined to reconstruct the table. This enables the recognition and reconstruction of both wired and wireless tables, improving the accuracy and robustness of table recognition.

[0115] Please refer to the following. Figure 3 This is a flowchart illustrating a table recognition method provided in an exemplary embodiment of this specification. Figure 3 As shown, the table recognition method may include the following steps:

[0116] S302, Obtain the target image containing the table.

[0117] Specifically, S302 is the same as S202, and will not be repeated here.

[0118] S304, determine the multimodal information corresponding to each target text in the table based on the target text in the target image, the target position corresponding to the target text, and the target image.

[0119] Specifically, in order to ensure the semantic relevance between target texts in each target cell of the reconstructed table, the positional relevance between target texts and target cells in the table, and the image feature relevance, the aforementioned multimodal information includes text modal information, positional modal information, and image feature modal information corresponding to the target texts. This allows for a more accurate determination of the global relationship information of the table based on the multimodal information corresponding to each target text, avoiding the impact of single modal information on table recognition and reconstruction, and improving the robustness and accuracy of table recognition.

[0120] Specifically, after obtaining the target image containing the table to be recognized, the above target image can be first subjected to text recognition to obtain the target text in the above table to be recognized and the target positions corresponding to each target text. Then, based on the target text and the target positions corresponding to the target text in the target image, the target image features corresponding to the target positions of the target text are extracted from the target image, and the text modality information corresponding to the target text is determined based on the target text in the above target image, the position modality information corresponding to the target text is determined based on the target position corresponding to the above target text, and the image feature modality information corresponding to the target text is determined based on the target image features corresponding to the above target text, that is, the target text, the target positions corresponding to the target text, and the target image features in the above table are respectively encoded into corresponding target text vectors (text modality information), target position vectors (position modality information), and target image feature vectors (image feature modality information), so as to obtain the multimodal information corresponding to each target text in the above table.

[0121] Exemplarily, as Figure 4 shown, after obtaining the target image 410 containing the table to be recognized, the text modality information 440 corresponding to each target text can be determined according to the target text (gold, amount, 1, 9,., full, etc.) of the table to be recognized in the target image 410, the position modality information 430 corresponding to each target text can be determined according to the target positions corresponding to each target text in the above table, and the image feature modality information 420 can be determined according to the target image features corresponding to the target positions of each target text in the above table.

[0122] S306. Determine the global relationship information corresponding to the table based on the multimodal information.

[0123] Specifically, in order to more fully挖掘 the global relationship information in the table and improve the accuracy of table recognition, after determining the multimodal information corresponding to each target text in the table, the multimodal information corresponding to each target text in the above table can be input into the encoder, and the target feature information corresponding to the table is output. The above target feature information includes the first target multimodal feature information of the multimodal information corresponding to each target text in the table and the global relationship information corresponding to the table. That is, the encoder can be used to encode the multimodal information corresponding to each target text in the table, so as to obtain the feature representation of the multimodal information of each target text in the upper space, that is, the first target multimodal feature information, and the global relationship information corresponding to the table. The above first target multimodal feature information can be the feature tensor corresponding to the target text in the upper space. The above encoder can be but is not limited to a BERT transform encoder, a node encoder, etc.

[0124] S308. Determine the local relationship information corresponding to the table based on the global relationship information.

[0125] Specifically, S308 is the same as S206, and will not be repeated here.

[0126] Specifically, the aforementioned local relationship information may include the target positional relationships between target text within each first region of the table and the target row and column relationships between target cells within each second region of the table. For example... Figure 5 As shown, the process of determining the local relationship information corresponding to a table based on global relationship information can include the following steps:

[0127] S502, determine the target position relationship based on target feature information.

[0128] Specifically, the aforementioned target feature information includes the first target multimodal feature information corresponding to the multimodal information of each target text in the table, and the global relationship information corresponding to the table. After obtaining the target feature information corresponding to the table using the encoder, the target positional relationship between the target texts in each first region of the table can be determined based on the aforementioned target feature information. The aforementioned target positional relationship is used to characterize whether each target text in each first region of the table is located in the same cell. For example... Figure 6 As shown, the process of determining the target position relationship in S502 above may include the following steps:

[0129] S602, Construct a target single-character sub-image set based on target feature information.

[0130] Specifically, the aforementioned target character sub-image set includes at least one target character sub-image, which includes multiple target text nodes located in the same area of ​​the table and the connection relationships between these nodes. Each target text node includes the first target multimodal feature information corresponding to that target text. The target text node set comprising each target character sub-image in the aforementioned target character sub-image set includes the target text nodes corresponding to all target texts in the table. The aforementioned first area can be the area in the table corresponding to the target texts corresponding to all target text nodes included in the target character sub-image.

[0131] Optionally, to reduce the complexity of constructing target character subgraphs and decrease the time required for table recognition, multiple target character subgraphs can be constructed for the entire table. Then, the target positional relationships between the target text nodes in each target character subgraph are calculated separately, thereby reducing the computational complexity and improving the efficiency of table recognition. The multiple target text nodes located in the same area (first area) of the table included in each of the aforementioned target character subgraphs may include, but are not limited to, a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node. The aforementioned target positional relationships are used to characterize the positional relationship between the central target text node and the first and second target text nodes, i.e., whether the central target text node, the first target text node, and the second target text node are located within the same cell.

[0132] Furthermore, after obtaining the first target multimodal feature information corresponding to each target character in the table to be recognized and the global relation information corresponding to the table using the encoder, at least one target character can be selected from each target character in the table as the central target character node corresponding to each target character subgraph. Then, the target characters adjacent to the central target character node, that is, the first target distance between the first target multimodal feature information of the central target character node and the target characters are less than or equal to the first distance, are taken as the first target character nodes corresponding to the central target character node. The target characters adjacent to the first target character node, that is, the second target distance between the first target multimodal feature information of the first target character node and the target characters are less than or equal to the second distance, are taken as the second target character nodes corresponding to the central target character node. Finally, the first target character node and its corresponding second target character node are connected with the central target character node as the center, thereby constructing the target character subgraph corresponding to the central target character node. The aforementioned first target distance is used to characterize the distance between the first target multimodal feature information corresponding to the central target text node and the first target multimodal feature information corresponding to other target text nodes besides the central target text node. The aforementioned second target distance is used to characterize the distance between the first target multimodal feature information corresponding to the first target text node and the first target multimodal feature information corresponding to other target text nodes besides the central target text node. The aforementioned distance can be, but is not limited to, Euclidean distance, etc. The aforementioned first distance can be less than or equal to the aforementioned second distance, or it can be greater than the aforementioned second distance. This specification does not limit this in the embodiments.

[0133] For example, such as Figure 7As shown, the aforementioned single-character subgraph may include a central target character node 710, six first target character nodes 720 adjacent to the central target character node 710, and five second target character nodes 730 adjacent to the first target character nodes 720. The distance between the six first target character nodes 720 and the central target character node 710 is less than or equal to a first distance 740, that is, the six first target character nodes 720 are adjacent to the central target character node 710; the distance between two second target character nodes 731 and second target character node 721 among the five second target character nodes 730 is less than or equal to a second distance 750, that is, the two second target character nodes 731 are adjacent to the second target character node 721; the distance between three second target character nodes 732 and second target character node 722 among the five second target character nodes 730 is less than or equal to a second distance 750, that is, the three second target character nodes 732 are adjacent to the second target character node 722.

[0134] Understandably, the aforementioned target character subgraph may include, in addition to a central target character node, a first target character node adjacent to the central target character node, and a second target character node adjacent to the first target character node, a third target character node adjacent to the second target character node, etc. This specification does not limit this aspect in the embodiments. When constructing a target character subgraph set based on target feature information, without considering time consumption and computational complexity, a target character subgraph can be directly constructed based on the target feature information; that is, the target character subgraph includes the target character nodes corresponding to all target characters in the table to be identified.

[0135] S604: Input the target single-character sub-image set into the first graph convolutional neural network, and output the target positional relationship between each target character node in the target single-character sub-image.

[0136] Specifically, after obtaining the target character sub-image set corresponding to the table to be identified, each target character sub-image in the above target character sub-image set can be input into the trained first graph convolutional neural network. The first graph convolutional neural network then performs inference to obtain the target positional relationships between the target character nodes in each of the above target character sub-images. The above first graph convolutional neural network is trained based on multiple character sub-images with known positional relationships between character nodes.

[0137] In this embodiment of the specification, after obtaining the first target multimodal feature information corresponding to the multimodal information of each target character in the table and the global relationship information corresponding to the table using the encoder, a target single-character sub-image set can be constructed based on the first target multimodal feature information corresponding to each target character and the global relationship information corresponding to the table. Then, the target positional relationship between each target character in each target single-character sub-image (first region) corresponding to the table is extracted by the trained first-image convolutional neural network, that is, the local relationship information between each target character in the table. Thus, without the need for line recognition, it is possible to know which target characters should be located in the same target cell, thereby improving the robustness of table recognition.

[0138] like Figure 5 As shown, in S502 above, after determining the target position relationship based on the target feature information, the implementation process of determining the local relationship information corresponding to the table based on the global relationship information further includes:

[0139] S504, determine the row and column relationship of the target based on the target position relationship and the multimodal feature information of the first target.

[0140] In this embodiment of the specification, when determining the local relationship information corresponding to a table based on global relationship information, the target positional relationship of the table can be determined first based on the target feature information, including the global relationship information corresponding to the table. That is, the local relationship information between target text in the table, such as, but not limited to, whether two adjacent target texts or target texts within a certain area are located in the same target cell. Then, based on the aforementioned local relationship information (target positional relationship) between target texts, the target row and column relationship of the table can be further determined, that is, the local relationship information between target cells in the table, such as, but not limited to, whether two adjacent target cells or target cells within a certain area belong to the same row or column. Thus, by combining the global relationship information in the table, the local relationship information in the table can be fully explored, improving the recognition rate and robustness of the table.

[0141] Specifically, such as Figure 8 As shown, the process of determining the target row-column relationship in S504 above may include the following steps:

[0142] S802, construct a target cell sub-map set based on the target positional relationship and the multimodal feature information of the first target.

[0143] Specifically, the aforementioned target cell subgraph set includes at least one target cell subgraph. Each target cell subgraph includes multiple target cell nodes located within the same area of ​​the table, as well as the connection relationships between these multiple target cell nodes. Each target cell node includes second target multimodal feature information corresponding to the target cell, which is obtained based on first target multimodal feature information corresponding to all target text located in that target cell node (target cell). The target cell node set comprising each target cell subgraph in the aforementioned target cell subgraph set includes the target cell nodes corresponding to all target cells in the table.

[0144] Specifically, such as Figure 9 As shown, the process of constructing the target cell sub-map set in S802 above can include the following steps:

[0145] S902, determine the target cell information corresponding to each target cell in the table based on the target position relationship.

[0146] Specifically, the target cell information mentioned above includes the target text located within the target cell. After determining the target positional relationship between the target text in each area of ​​the table, it is possible to further determine which target text in the table is located in the same target cell based on the above target positional relationship, that is, which target text (target cell information) is included in each target cell of the table.

[0147] For example, if the target position relationship (local relationship information) between target texts in the table indicates that target text a and target text b in region A are located in the same target cell K, and target text b and target text c in region B are also located in the same target cell K, then it can be determined that target text a, target text b, and target text c in the table are all located in target cell K in the table based on the above target position relationship. Thus, the target cell information corresponding to the above target cell K can be obtained, that is, target cell K includes target text a, target text b, and target text c.

[0148] S904, fuse the first target multimodal feature information corresponding to each target text in the target cell to obtain the second target multimodal feature information.

[0149] Specifically, after determining the target cell information corresponding to each target cell in the table, the first target multimodal feature information corresponding to each target text in the target cell can be further fused based on the target cell information to obtain the second target multimodal feature information corresponding to each target cell in the table.

[0150] S906, construct target cell sub-map set based on the multimodal feature information of the second target.

[0151] Specifically, after obtaining the second target multimodal feature information corresponding to each target cell in the table, a target cell sub-map set can be constructed based on the distance between the second target multimodal feature information corresponding to each target cell. The above-described process of constructing the target cell sub-map set is similar to the process of constructing the target single-character sub-map set in S602, and will not be repeated here.

[0152] Optionally, after determining the target positional relationship between the target characters within each first region of the table, a cell diagram can be generated based on the target characters located within the same target cell in the table. The nodes in this cell diagram are the target character nodes corresponding to the target characters located within the same target cell. These target character nodes include the first target multimodal feature information corresponding to the target character. That is, the target character sub-graphs can be merged to form a new cell diagram based on the target positional relationship between the target character nodes (target characters) in each target character sub-graph of the table. The number of cell diagrams can correspond to the number of regions corresponding to each target character in the table to be identified. After obtaining the cell diagrams corresponding to the table to be identified, each cell diagram can be used as a target cell node, and then... Figure 6 The target cell subgraph set corresponding to the table is constructed in a similar manner to the target single-character subgraph in S602 shown. The second target multimodal feature information included in the target cell node of the above target cell subgraph is obtained by fusing the first target multimodal feature information corresponding to all target characters located in the target cell node (target cell).

[0153] Optionally, to reduce the complexity of constructing the target cell subgraph and decrease the time required for table recognition, multiple target cell subgraphs can be constructed for the entire table. Then, the target row and column relationships between the target cell nodes in each target cell subgraph are calculated separately, thereby further reducing the computational complexity and improving the efficiency of table recognition. The multiple target cell nodes located in the same area of ​​the table included in the aforementioned target cell subgraph may include a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node. The central target cell node is located at the center of the target cell subgraph. The aforementioned target row and column relationships are used to characterize the row and column relationships between the central target cell node and the first and second target cell nodes.

[0154] Understandably, the aforementioned target cell subgraph may include, in addition to a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node, a third target cell node adjacent to the second target cell node, etc. This specification does not limit this. When constructing a target cell subgraph set based on target positional relationships and first target multimodal feature information, without considering time consumption and computational complexity, a target cell subgraph can be directly constructed based on target positional relationships, i.e., target text located in the same target cell and the corresponding first target multimodal feature information. That is, the target cell subgraph includes the target cell nodes corresponding to all target cells in the table to be identified.

[0155] S804: Input the target cell subgraph set into the second graph convolutional neural network, and output the target row and column relationships between each target cell node in the target cell subgraph.

[0156] Specifically, after obtaining the target cell subgraph set corresponding to the table to be identified, each target cell subgraph in the above target cell subgraph set can be input into the trained second-graph convolutional neural network. The second-graph convolutional neural network then infers the target row and column relationships between the target cell nodes in each of the above target cell subgraphs. The above-mentioned second-graph convolutional neural network is trained based on multiple cell subgraphs with known row and column relationships between cell nodes.

[0157] In this embodiment of the specification, after determining the target positional relationship (local relationship information) between target characters in each first region of the table based on the target feature information, a target cell sub-image set can be constructed based on the first target multimodal feature information corresponding to each target character and the target positional relationship (local relationship information) between target characters in each region of the table. Then, the target row and column relationship between each target cell in each target cell sub-image (second region) corresponding to the table is extracted by the trained second graph convolutional neural network, that is, the local relationship information between each target cell in the table. Thus, without the need for line recognition, it is possible to know which target cells in the table to be recognized are in the same row and which target cells are in the same column, thereby improving the robustness of table recognition.

[0158] like Figure 3 As shown in S308, after determining the local relationship information corresponding to the table based on the global relationship information, the above table recognition method further includes:

[0159] S310, Reconstruct the table in the target image based on local relational information.

[0160] Optionally, the table in the target image can be reconstructed based on the target positional relationships and target row-column relationships in the table. That is, the target text located in the same target cell can be determined first from the target positional relationships (local relationship information) between the target text in each region of the table. In other words, the target cell information is determined first based on the target positional relationships in the table to be identified. Then, the target cell information is determined from the target row-column relationships (local relationship information) between the target cells in each region of the table. Finally, the table is reconstructed based on the target row-column relationships and target cell information. Thus, it is not necessary to perform line recognition on the table. Whether it is a wired table or a wireless table, the table recognition and reconstruction can be accurately achieved.

[0161] Optionally, after determining the target row-column relationship (local relationship information) between target cells in the table, a target row atlas and a target column atlas can be generated based on this target row-column relationship and the target cell sub-atlas used to determine it. Then, the table in the target image can be reconstructed based on the aforementioned target row atlas, target column atlas, and the target cell information corresponding to the target cell nodes. The aforementioned target row atlas includes the target row atlas corresponding to each row in the table, and each target row atlas includes the target cell node corresponding to the target cell in the same row. The aforementioned target column atlas includes the target column atlas corresponding to each column in the table, and each target column atlas includes the target cell node corresponding to the target cell in the same column.

[0162] Please refer to the following. Figure 10 This is a schematic diagram illustrating the implementation process of a table recognition method provided in an exemplary embodiment of this specification. Figure 10As shown, after obtaining the target image containing the table to be recognized, the target image can first be input into the preprocessing module for preprocessing (e.g., but not limited to image deblurring, image brightness enhancement, image contrast enhancement, image super-resolution reconstruction, etc.) to enhance the quality of the target image and improve the table recognition effect. Then, the preprocessed target image is input into the image correction module to perform large-direction and small-angle corrections to further improve the table recognition effect. Next, a text recognition module, such as but not limited to optical character recognition (OCR), is used to perform text recognition on the corrected target image to obtain the target text in the table to be recognized and the target positions corresponding to each target text. The target text can include, but is not limited to, characters, etc. Then, the Faster R-CNN algorithm can be used to extract the target image features corresponding to the target positions of each target text from the target image, and the target text, target positions, and target image features are converted into corresponding vectors to obtain the multimodal information corresponding to each target text. This multimodal information is then input into the encoder to output the target feature information corresponding to the table to be recognized. The target feature information includes the feature representation of the multimodal information of the aforementioned target texts in the upper-level space, namely the first target multimodal feature information, and the global relationship information corresponding to the table to be identified. Then, according to... Figure 6 The method described in S602 constructs a target character sub-image set corresponding to the table to be recognized based on the aforementioned target feature information, and inputs the aforementioned target character sub-image set into a trained first-graph convolutional neural network to output the target positional relationship between target characters within each target character sub-image in the table to be recognized. Then, the target positional relationships in the table to be recognized can be fused to obtain the target cell information corresponding to each target cell, i.e., which target characters are included in the target cell. Based on the target cell information corresponding to each target cell in the table to be recognized and the aforementioned target feature information, a target cell sub-image set is constructed in a similar manner to when constructing the target character sub-image set. After constructing the target cell sub-image set corresponding to the table to be recognized, the target cell sub-image set can be input into a trained second-graph convolutional neural network to output the target row and column relationships between target cells in each target cell sub-image. Finally, the table (the table to be recognized) is reconstructed based on the target positional relationships (local relationship information) between target characters and the target row and column relationships (local relationship information) between target cells in the table to be recognized. The reconstructed table can be output in file formats such as Excel and XML, but is not limited to those specified.

[0163] Please refer to Figure 11 , Figure 11 This specification provides an exemplary embodiment of a form recognition device. For example... Figure 11 As shown, the form recognition device 1100 includes:

[0164] Module 1110 is used to acquire the target image containing the table;

[0165] The first determining module 1120 is used to determine the global relationship information corresponding to the table based on the target image; the global relationship information includes the relationship between each target text in the table and the relationship between each target cell.

[0166] The second determining module 1130 is used to determine the local relationship information corresponding to the table based on the global relationship information mentioned above; the local relationship information includes the relationship between each target text in each first area of ​​the table and the relationship between each target cell in each second area of ​​the table.

[0167] The reconstruction module 1140 is used to reconstruct the table in the target image based on the aforementioned local relationship information.

[0168] In one possible implementation, the first determining module 1120 includes:

[0169] The first determining unit is used to determine the multimodal information corresponding to each target character in the table based on the target characters in the target image, the target positions corresponding to the target characters, and the target image.

[0170] The second determining unit is used to determine the global relationship information corresponding to the above table based on the above multimodal information.

[0171] In one possible implementation, the aforementioned multimodal information includes text modal information, location modal information, and image feature modal information corresponding to the target text.

[0172] The aforementioned first determining unit includes:

[0173] An extraction subunit is used to extract the target image features corresponding to the target text from the target image based on the target text in the target image and the target position corresponding to the target text.

[0174] The determination subunit is used to determine the text modality information based on the target text in the target image, to determine the position modality information based on the target position corresponding to the target text, and to determine the image feature modality information based on the target image features corresponding to the target text.

[0175] In one possible implementation, the second determining unit is specifically used for:

[0176] The multimodal information corresponding to each target character in the above table is input into the encoder, and the target feature information corresponding to the above table is output. The target feature information includes the first target multimodal feature information of the multimodal information corresponding to each target character in the above table and the global relationship information corresponding to the above table.

[0177] In one possible implementation, the aforementioned local relationship information includes the target positional relationship between target texts in each first region of the table and the target row and column relationship between target cells in each second region of the table.

[0178] The second determining module 1130 mentioned above includes:

[0179] The third determining unit is used to determine the target positional relationship based on the target feature information mentioned above; the target positional relationship is used to characterize whether each target text in each first area of ​​the table is located in the same cell.

[0180] The fourth determining unit is used to determine the row and column relationship of the target based on the above target position relationship and the above target multimodal feature information.

[0181] In one possible implementation, the third determining unit mentioned above includes:

[0182] The first construction subunit is used to construct a target single-character sub-graph set based on the aforementioned target feature information; the aforementioned target single-character sub-graph set includes at least one target single-character sub-graph; the aforementioned target single-character sub-graph includes multiple target text nodes located in the same area of ​​the aforementioned table and the connection relationship between the multiple target text nodes; the aforementioned target text node includes the first target multimodal feature information corresponding to the aforementioned target text;

[0183] The first convolutional sub-unit is used to input the above target single-character sub-graph set into the first graph convolutional neural network and output the target positional relationship between each target character node in the above target single-character sub-graph; the above first graph convolutional neural network is trained based on multiple single-character sub-graphs with known positional relationships between each character node.

[0184] In one possible implementation, the fourth determining unit mentioned above includes:

[0185] The second construction subunit is used to construct a target cell sub-graphet based on the aforementioned target positional relationships and the aforementioned first target multimodal feature information; the aforementioned target cell sub-graphet includes at least one target cell sub-graph; the aforementioned target cell sub-graph includes multiple target cell nodes located in the same area of ​​the aforementioned table and the connection relationships between the multiple target cell nodes; the aforementioned target cell node includes the second target multimodal feature information corresponding to the aforementioned target cell; the aforementioned second target multimodal feature information is obtained based on the first target multimodal feature information corresponding to all target text located in the aforementioned target cell node;

[0186] The second convolutional subunit is used to input the above target cell subgraph set into the second graph convolutional neural network and output the target row and column relationship between each target cell node in the above target cell subgraph; the above second graph convolutional neural network is trained based on multiple cell subgraphs with known row and column relationships between each cell node.

[0187] In one possible implementation, the second building block described above is specifically used for:

[0188] Based on the above target positional relationships, the target cell information corresponding to each target cell in the above table is determined; the target cell information includes the target text located in the target cell; the first target multimodal feature information corresponding to each target text in the target cell is fused to obtain the second target multimodal feature information; and a target cell sub-atlas is constructed based on the second target multimodal feature information.

[0189] In one possible implementation, the plurality of target text nodes includes a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node; the target positional relationship is used to characterize the positional relationship between the central target text node and the first and second target text nodes.

[0190] In one possible implementation, the plurality of target cell nodes include a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node; the target row-column relationship is used to characterize the row-column relationship between the central target cell node and the first and second target cell nodes.

[0191] In one possible implementation, the aforementioned refactoring module 1140 is specifically used for:

[0192] Reconstruct the table in the target image based on the target positional relationships and target row-column relationships in the table above.

[0193] In one possible implementation, the aforementioned refactoring module 1140 includes:

[0194] The generation unit is used to generate a target row map set and a target column map set based on the target row-column relationship and the target cell sub-map set. The target row map set includes the target row map corresponding to each row in the table. The target row map includes the target cell node corresponding to the target cell in the same row. The target column map set includes the target column map corresponding to each column in the table. The target column map includes the target cell node corresponding to the target cell in the same column.

[0195] The reconstruction unit is used to reconstruct the table in the target image based on the target row atlas, the target column atlas, and the target cell information corresponding to the target cell node.

[0196] The division of modules in the above-described form recognition device is for illustrative purposes only. In other embodiments, the form recognition device can be divided into different modules as needed to complete all or part of the functions of the above-described form recognition device. The implementation of each module in the form recognition device provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the form recognition method described in the embodiments of this specification.

[0197] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. For example... Figure 12 As shown, the electronic device 1200 may include: at least one processor 1210, at least one communication bus 1220, a user interface 1230, at least one network interface 1240, and a memory 1250. The communication bus 1220 can be used to enable communication between the aforementioned components.

[0198] The user interface 1230 may include a display screen and a camera. Optionally, the user interface 1230 may also include a standard wired interface and a wireless interface.

[0199] The network interface 1240 may optionally include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, etc.

[0200] The processor 1210 may include one or more processing cores. The processor 1210 connects to various parts within the electronic device 1200 using various interfaces and lines. It executes various functions and processes data of the routing electronic device 1200 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1250, and by calling data stored in the memory 1250. Optionally, the processor 1210 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1210 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 1210 and may be implemented as a separate chip.

[0201] The memory 1250 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1250 may include a non-transitory computer-readable medium. The memory 1250 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1250 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as fetching, determining, refactoring, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 1250 may also be at least one storage device located remotely from the aforementioned processor 1210. Figure 12 As shown, the memory 1250, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0202] Specifically, the processor 1210 can be used to call the application stored in the memory 1250 and specifically perform the following operations: acquire a target image containing a table; determine global relationship information corresponding to the table based on the target image; the global relationship information includes the relationship between each target text in the table and the relationship between each target cell; determine local relationship information corresponding to the table based on the global relationship information; the local relationship information includes the relationship between each target text in each first region of the table and the relationship between each target cell in each second region of the table; and reconstruct the table in the target image based on the local relationship information.

[0203] In some possible embodiments, when the processor 1210 executes the determination of global relationship information corresponding to the table based on the target image, it is specifically used to perform: determining multimodal information corresponding to each target character in the table based on the target text in the target image, the target position corresponding to the target text, and the target image; and determining global relationship information corresponding to the table based on the multimodal information.

[0204] In some possible embodiments, the multimodal information includes text modal information, position modal information, and image feature modal information corresponding to the target text. When the processor 1210 executes the determination of the multimodal information corresponding to each target text in the table based on the target text in the target image, the target position corresponding to the target text, and the target image, it is specifically used to perform the following: extracting the target image features corresponding to the target text from the target image based on the target text in the target image and the target position corresponding to the target text; determining the text modal information based on the target text in the target image, determining the position modal information based on the target position corresponding to the target text, and determining the image feature modal information based on the target image features corresponding to the target text.

[0205] In some possible embodiments, when the processor 1210 executes the determination of the global relationship information corresponding to the table based on the multimodal information, it is specifically used to: input the multimodal information corresponding to each target character in the table into the encoder, and output the target feature information corresponding to the table; the target feature information includes the first target multimodal feature information of the multimodal information corresponding to each target character in the table and the global relationship information corresponding to the table.

[0206] In some possible embodiments, the aforementioned local relationship information includes the target positional relationship between target texts in each first region of the table and the target row and column relationship between target cells in each second region of the table;

[0207] When the processor 1210 executes the determination of the local relationship information corresponding to the table based on the global relationship information, it is specifically used to perform: determining the target position relationship based on the target feature information; the target position relationship is used to characterize whether each target text in each first region of the table is located in the same cell; and determining the target row and column relationship based on the target position relationship and the first target multimodal feature information.

[0208] In some possible embodiments, when the processor 1210 executes the determination of the target positional relationship based on the target feature information, it is specifically used to perform the following: constructing a target single-character sub-graph set based on the target feature information; the target single-character sub-graph set includes at least one target single-character sub-graph; the target single-character sub-graph includes multiple target text nodes located in the same area of ​​the table and the connection relationship between the multiple target text nodes; the target text node includes first target multimodal feature information corresponding to the target text; inputting the target single-character sub-graph set into a first graph convolutional neural network, and outputting the target positional relationship between each target text node in the target single-character sub-graph; the first graph convolutional neural network is trained based on multiple single-character sub-graphs with known positional relationships between each text node.

[0209] In some possible embodiments, when the processor 1210 executes the determination of the target row-column relationship based on the target positional relationship and the first target multimodal feature information, it is specifically used to perform the following: constructing a target cell sub-graph set based on the target positional relationship and the first target multimodal feature information; the target cell sub-graph set includes at least one target cell sub-graph; the target cell sub-graph includes multiple target cell nodes located in the same area of ​​the table and the connection relationship between the multiple target cell nodes; the target cell node includes the second target multimodal feature information corresponding to the target cell; the second target multimodal feature information is obtained based on the first target multimodal feature information corresponding to all target text located in the target cell node; inputting the target cell sub-graph set into a second graph convolutional neural network, and outputting the target row-column relationship between each target cell node in the target cell sub-graph; the second graph convolutional neural network is trained based on multiple cell sub-graphs with known row-column relationships between each cell node.

[0210] In some possible embodiments, when the processor 1210 executes the construction of a target cell sub-map based on the target positional relationship and the first target multimodal feature information, it is specifically used to perform: determining the target cell information corresponding to each target cell in the table based on the target positional relationship; the target cell information includes the target text located in the target cell; fusing the first target multimodal feature information corresponding to each target text in the target cell to obtain the second target multimodal feature information; and constructing a target cell sub-map based on the second target multimodal feature information.

[0211] In some possible embodiments, the plurality of target text nodes include a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node; the target positional relationship is used to characterize the positional relationship between the central target text node and the first and second target text nodes.

[0212] In some possible embodiments, the plurality of target cell nodes include a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node; the target row-column relationship is used to characterize the row-column relationship between the central target cell node and the first and second target cell nodes.

[0213] In some possible embodiments, when the processor 1210 reconstructs the table in the target image based on the local relation information, it specifically performs the following:

[0214] Reconstruct the table in the target image based on the target positional relationships and target row-column relationships in the table above.

[0215] In some possible embodiments, when the processor 1210 reconstructs the table in the target image based on the local relationship information, it specifically performs the following: generating a target row map set and a target column map set based on the target row and column relationships and the target cell sub-map set; the target row map set includes the target row map corresponding to each row in the table; the target row map includes the target cell node corresponding to the target cell in the same row; the target column map set includes the target column map corresponding to each column in the table; the target column map includes the target cell node corresponding to the target cell in the same column; and reconstructs the table in the target image based on the target row map set, the target column map set, and the target cell information corresponding to the target cell node.

[0216] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described form recognition device are implemented as software functional units and sold or used as independent products, they can be stored in the above-described computer-readable storage medium.

[0217] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0218] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0219] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims.

[0220] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims and specification may be performed in a different order than in the embodiments described in the specification and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A table recognition method, the method comprising: Get the target image containing the table; Determine the global relationship information corresponding to the table based on the target image; The global relationship information includes the relationships between target texts in the table and the relationships between target cells; Based on the global relationship information, the local relationship information corresponding to the table is determined; the local relationship information includes the target positional relationship between each target text in each first area of ​​the table and the target row and column relationship between each target cell in each second area of ​​the table; Reconstruct the table in the target image based on the local relation information; Determining the local relationship information corresponding to the table based on the global relationship information includes: The target location relationship is determined based on the target feature information corresponding to the table; the target feature information includes the global relationship information. The target row and column relationship is determined based on the target positional relationship.

2. The method as described in claim 1, wherein determining the global relationship information corresponding to the table based on the target image includes: Based on the target text in the target image, the target position corresponding to the target text, and the target image, determine the multimodal information corresponding to each target text in the table; The global relationship information corresponding to the table is determined based on the multimodal information.

3. The method as described in claim 2, wherein the multimodal information includes text modal information, positional modal information, and image feature modal information corresponding to the target text; The step of determining the multimodal information corresponding to each target character in the table based on the target text in the target image, the target position corresponding to the target text, and the target image includes: Based on the target text in the target image and the target position corresponding to the target text, extract the target image features corresponding to the target text from the target image; The text modality information is determined based on the target text in the target image, the position modality information is determined based on the target position corresponding to the target text, and the image feature modality information is determined based on the target image features corresponding to the target text.

4. The method as described in claim 2, wherein determining the global relationship information corresponding to the table based on the multimodal information includes: The multimodal information corresponding to each target text in the table is input into the encoder, and the target feature information corresponding to the table is output. The target feature information includes the first target multimodal feature information corresponding to the multimodal information of each target text in the table, and the global relationship information corresponding to the table.

5. The method as described in claim 4, wherein determining the local relation information corresponding to the table based on the global relation information includes: The target positional relationship is determined based on the target feature information; The target position relationship is used to characterize whether each target text in each first area of ​​the table is located in the same cell; The target row and column relationship is determined based on the target position relationship and the first target multimodal feature information.

6. The method of claim 5, wherein determining the target position relationship based on the target feature information comprises: Construct a target single-character sub-image set based on the target feature information; The target single-character sub-graph set includes at least one target single-character sub-graph; The target single-character sub-image includes multiple target text nodes located in the same area of ​​the table and the connection relationships between the multiple target text nodes; the target text node includes the first target multimodal feature information corresponding to the target text; The target single-character sub-graph set is input into the first graph convolutional neural network, which outputs the target positional relationship between each target character node in the target single-character sub-graph; the first graph convolutional neural network is trained based on multiple single-character sub-graphs with known positional relationships between each character node.

7. The method of claim 5, wherein determining the target row-column relationship based on the target positional relationship and the first target multimodal feature information comprises: Based on the target location relationships and the first target multimodal feature information, a target cell sub-map set is constructed; The target cell sub-map set includes at least one target cell sub-map; The target cell subgraph includes multiple target cell nodes located in the same area of ​​the table and the connection relationships between the multiple target cell nodes; the target cell node includes the second target multimodal feature information corresponding to the target cell; The second target multimodal feature information is obtained based on the first target multimodal feature information corresponding to all target text located in the target cell node; The target cell subgraph set is input into the second graph convolutional neural network, which outputs the target row and column relationships between each target cell node in the target cell subgraph. The second graph convolutional neural network is trained based on multiple cell subgraphs with known row and column relationships between each cell node.

8. The method of claim 7, wherein constructing the target cell sub-map set based on the target positional relationship and the first target multimodal feature information comprises: Based on the target positional relationship, determine the target cell information corresponding to each target cell in the table; The target cell information includes the target text located in the target cell; The second target multimodal feature information is obtained by fusing the first target multimodal feature information corresponding to each target text in the target cell; A target cell sub-map set is constructed based on the second target multimodal feature information.

9. The method of claim 6, wherein the plurality of target text nodes includes a central target text node, a first target text node adjacent to the central target text node, and a second target text node adjacent to the first target text node; the target positional relationship is used to characterize the positional relationship between the central target text node and the first target text node and the second target text node.

10. The method of claim 7, wherein the plurality of target cell nodes includes a central target cell node, a first target cell node adjacent to the central target cell node, and a second target cell node adjacent to the first target cell node; the target row-column relationship is used to characterize the row-column relationship between the central target cell node and the first target cell node and the second target cell node.

11. The method of claim 5, wherein reconstructing the table in the target image based on the local relation information comprises: Reconstruct the table in the target image based on the target positional relationships and target row-column relationships in the table.

12. The method of claim 8, wherein reconstructing the table in the target image based on the local relation information comprises: Based on the target row and column relationships and the target cell sub-atlas, a target row atlas and a target column atlas are generated; the target row atlas includes the target row atlas corresponding to each row in the table; the target row atlas includes the target cell node corresponding to the target cell located in the same row; the target column atlas includes the target column atlas corresponding to each column in the table; the target column atlas includes the target cell node corresponding to the target cell located in the same column. The table in the target image is reconstructed based on the target row atlas, the target column atlas, and the target cell information corresponding to the target cell node.

13. A form recognition device, the device comprising: The acquisition module is used to acquire the target image containing the table; The first determining module is used to determine the global relationship information corresponding to the table based on the target image; The global relationship information includes the relationships between target texts in the table and the relationships between target cells; The second determining module is used to determine the local relationship information corresponding to the table based on the global relationship information; The local relationship information includes the target positional relationship between each target text in each first area of ​​the table and the target row and column relationship between each target cell in each second area of ​​the table; The reconstruction module is used to reconstruct the table in the target image based on the local relationship information; The second determining module is specifically used for: The target location relationship is determined based on the target feature information corresponding to the table; the target feature information includes the global relationship information. The target row and column relationship is determined based on the target positional relationship.

14. An electronic device comprising: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-12.

15. A computer storage medium storing a plurality of instructions adapted for loading by a processor and performing the method steps of any one of claims 1-12.

16. A computer program product comprising instructions that, when run on a computer or processor, causes the computer or processor to perform the table recognition method as described in any one of claims 1-12.

Images