Table type determination method, device and medium based on neural network model

Abstract

This application relates to the field of artificial intelligence technology, and provides a method, system, device and storage medium for determining table types based on a neural network model. It includes: inputting the first form information into a correction network to obtain the first corrected result information, inputting it into a frame feature extraction network to obtain the first table line feature information and the first cell feature information; constructing a first positioning coordinate system with the first table line feature information, inputting the first cell feature information to generate the first cell positioning information, traversing the first cell set for row and column vector identification to obtain the first cell row vector and the first cell column vector, inputting them into a row and column feature extraction network to obtain the first semantic feature information and the first layout feature information, inputting them into a table type recognition network for table type optimization and screening to obtain the first screening result, and generating the first table type determination result. This application also relates to the field of blockchain technology, and the above table types can be stored in a node of a blockchain.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly relates to a method, system, device and storage medium for determining the type of a table based on a neural network model. Background Art

[0002] In many industries, such as the financial industry and the engineering industry, a large number of tables need to be faced. In the past, manual processing and formulation of tables were required. With the rapid development of artificial intelligence, due to the certain regularity of the tables themselves, they have received much attention. Among them, the recognition of the type of table is the basis for processing table data, and the determination of the type of table layout plays an important role in subsequent understanding and extraction logic.

[0003] In the prior art, the general determination of the type of table is mostly based on a rule system, or a shallow classifier trained based on manually designed features is used to identify the type of table. However, since the manually set classification rules rely on expert experience, there are problems that the transferability and accuracy are difficult to guarantee.

[0004] Therefore, how to obtain a table type recognition solution with better transferability and stable accuracy has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of the above, the present application provides a method, system, device and storage medium for determining the type of a table based on a neural network model, and the purpose is to obtain a table type recognition solution with better transferability and stable accuracy.

[0006] To achieve the above object, the present application provides a method for determining the type of a table based on a neural network model. The model includes a correction network, a frame feature extraction network, a row-column feature extraction network and a table type recognition network. The method includes:

[0007] Input the first form information into the correction network to obtain the first corrected result information;

[0008] Input the first corrected result information into the frame feature extraction network to obtain the first table frame feature information, where the first table frame feature information includes the first table line feature information and the first cell feature information;

[0009] Construct a first positioning coordinate system according to the first table line feature information, and input the first cell feature information into the first positioning coordinate system to generate the first cell positioning information;

[0010] Traverse the first cell set based on the first cell positioning information to perform row-column vector identification, and generate the first cell row vector and the first cell column vector;

[0011] Input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0012] Input the first semantic feature information and the first layout feature information into a table type recognition network for optimized screening of the table type to obtain a first screening result;

[0013] Generate a first table type determination result according to the first screening result.

[0014] Preferably, the step of inputting the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information includes:

[0015] Obtain a row vector feature extraction network and a column vector feature extraction network according to the row-column feature extraction network;

[0016] Input the first cell row vector into the row vector feature extraction network to obtain first row feature information;

[0017] Input the first cell column vector into the column vector feature extraction network to obtain first column vector feature information;

[0018] Perform row-column feature fusion on the first row feature information and the first column vector feature information to obtain the first row-column feature information.

[0019] Preferably, the step of performing row-column feature fusion on the first row feature information and the first column vector feature information to obtain the first row-column feature information includes:

[0020] Obtain a first linear layer and a second linear layer, where the first linear layer is used for semantic content matching and the second linear layer is used for semantic content classification;

[0021] Input the first row feature information into the first linear layer based on an attention mechanism to obtain a first semantic content matching result, where the first semantic content matching result includes first adjacent row semantic content;

[0022] Input the first column feature information into the first linear layer based on an attention mechanism to obtain a second semantic content matching result, where the second semantic content matching result includes first adjacent column semantic content;

[0023] Input the first adjacent row semantic content and the first adjacent column semantic content at the same coordinate position into the second linear layer to obtain first subject feature information, first object feature information, and first attribute feature information;

[0024] Add the first main feature information, the first object feature information, and the first attribute feature information into the first semantic feature information.

[0025] Preferably, the row-column feature fusion of the first row feature information and the first column vector feature information to obtain the first row-column feature information includes:

[0026] Obtain a third linear layer and a fourth linear layer, where the third linear layer is used for layout form matching, and the fourth linear layer is used for layout form classification;

[0027] Input the first row feature information into the third linear layer based on the attention mechanism to obtain a first layout form matching result, where the first layout form matching result includes the layout form information of adjacent cells in the first row;

[0028] Input the first column feature information into the third linear layer based on the attention mechanism to obtain a second layout form matching result, where the second layout form matching result includes the layout form information of adjacent cells in the first column;

[0029] Input the layout form information of adjacent cells in the first row and the layout form information of adjacent cells in the first column at the same coordinate position into the fourth linear layer to generate first layout form classification information;

[0030] Add the first layout form classification information into the first layout feature information.

[0031] Preferably, the input of the first semantic feature information and the first layout feature information into the table type recognition network for table type optimization and screening to obtain a first screening result includes:

[0032] Based on big data, perform table type matching according to the first layout feature information to obtain a first table type matching set;

[0033] Construct a first optimization and screening space according to the first table type matching set;

[0034] Input the first semantic feature information into the first optimization and screening space to obtain the first screening result.

[0035] Preferably, the construction of the first optimization and screening space according to the first table type matching set includes:

[0036] Obtain a first matching degree calculation formula:

[0037]

[0038] where P nRefers to the matching probability of the nth table type under the input semantic features, f 1n Indicates the selection frequency of the nth table type under the input semantic features in the local table type matching, f 2n Indicates the selection frequency of the nth table type under the input semantic features in the big data table type matching. α and β are custom proportional parameters representing the weights of f 1n and f 2n The number of table types is N;

[0039] Construct the first optimized screening space according to the first matching degree calculation formula and the first table type matching set.

[0040] Preferably, inputting the first semantic feature information into the first optimized screening space to obtain the first screening result includes:

[0041] Obtain the first constraint function:

[0042] When P n+1 -P n ≥0, T = 1

[0043] When P n+1 -P n <0,

[0044] where T represents the acceptance degree of transferring from the nth table type to the n + 1th table type;

[0045] When P n+1 -P n <0, when T ≥ the first preset acceptance degree, accept the n + 1th table type.

[0046] To implement the above method, the present application also provides a table type determination system based on a neural network model. The system includes:

[0047] The first obtaining unit is used to input the first form information into the correction network to obtain the first corrected result information;

[0048] The second obtaining unit is used to input the first corrected result information into the frame feature extraction network to obtain the first table frame feature information, where the first table frame feature information includes the first table line feature information and the first cell feature information;

[0049] The first generating unit is used to construct the first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate the first cell positioning information;

[0050] A second generation unit, configured to traverse the first cell set based on the first cell positioning information for row and column vector identification, and generate a first cell row vector and a first cell column vector;

[0051] A third acquisition unit, configured to input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0052] A fourth acquisition unit, configured to input the first semantic feature information and the first layout feature information into a table type recognition network for table type optimization and screening to obtain a first screening result;

[0053] A third generation unit, configured to generate a first table type determination result according to the first screening result.

[0054] To achieve the above object, the present application further provides an electronic device, where the electronic device includes:

[0055] At least one processor; and,

[0056] A memory communicatively connected to the at least one processor; where,

[0057] The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor can execute the table type determination method based on a neural network model described in any one of the above.

[0058] There is also provided a computer-readable storage medium, where the computer-readable storage medium stores a table type determination program based on a neural network model, and when the table type determination program based on the neural network model is executed by a processor, the steps of the table type determination method based on a neural network model described in any one of the above are implemented.

[0059] The table type determination method proposed in this application. The neural network model includes a correction network, a frame feature extraction network, a row-column feature extraction network, and a table type recognition network. The form information is corrected by the correction network, and then the corrected information is used for frame feature extraction to obtain table line features and cell features; a positioning coordinate system is constructed based on the table line features and cell features to obtain the positioning information of the cells; the cells are marked with row and column vectors, and then feature extraction is performed respectively based on the row vectors and column vectors, and the row vector and column vector feature information is analyzed to obtain semantic feature information and layout feature information; based on the semantic feature information and layout feature information, the table type recognition network is used for optimized screening of the table type to obtain the table type determination result. By separately extracting features from the row and column vectors, the semantic feature information and layout feature information obtained through fusion can represent the correlation between the semantic content and the layout, and then the table type recognition network is used for optimization based on the correlation relationship, obtaining a relatively accurate table type recognition result, and not relying on expert-set recognition rules, achieving the technical effects of improving the migration ability and judgment accuracy of the table type determination scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 FIG. is a schematic flowchart of a preferred embodiment of a table type determination method based on a neural network model provided by this application;

[0061] Figure 2 FIG. is a network schematic diagram of a preferred embodiment of the neural network model of this application;

[0062] Figure 3 FIG. is a schematic structural diagram of a preferred embodiment of a table type determination system based on a neural network model provided by this application;

[0063] Figure 4 FIG. is a schematic diagram of a preferred embodiment of an electronic device of this application;

[0064] The implementation, functional features, and advantages of the objectives of this application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] In order to make the objectives, technical solutions, and advantages of this application clearer, the following further details this application with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0066] This application provides a table type determination method based on a neural network model. Refer to Figure 1As shown in the figure, it is a schematic flowchart of the method of the embodiment of the table type determination method based on the neural network model in this application. This method can be executed by an electronic device, which can be implemented by software and / or hardware.

[0067] The neural network model refers to an intelligent model trained based on a neural network. When training and reasoning, the input data is form information, and the output information is the table determination result. Preferably, supervised learning is used for training. The neural network model mainly includes a correction network, a frame feature extraction network, a row and column feature extraction network, and a table type recognition network. Among them, the correction network is used to correct the angle and arrangement of information such as the table form and text content of the input form information; the frame feature extraction network is used to extract features of non-text content; the row and column feature extraction network is used to extract row features and column features of the table; the table type recognition network is used to identify and screen the table type.

[0068] The table type determination method based on the neural network model includes:

[0069] S10: Input the first form information into the correction network to obtain the first corrected result information;

[0070] Specifically, the first form information refers to the form information to be recognized as a table, and the input forms include but are not limited to: document form, photo, pdf, etc.; the first corrected result information refers to the form information after the correction work of the form information such as the table form and text content by inputting the first form information into the correction network.

[0071] Since the form is not limited to the table made by the table-making software on the computer, but also includes forms such as manually filled forms through taking pictures and manually made forms, etc., by setting the correction network to correct the text content and table in the first form information, it is convenient to more quickly identify the table type in the subsequent steps.

[0072] S20: Input the first corrected result information into the frame feature extraction network to obtain the first table frame feature information, and the first table frame feature information includes the first table line feature information and the first cell feature information;

[0073] Specifically, the first table frame feature information refers to the table feature information of non-text content. The frame feature extraction network is preferably constructed based on a convolutional neural network and obtained by training multiple groups of form information and corresponding frame feature identification information. When the first correction result information of the previous network is input into the frame feature extraction network, the generated first table frame feature information eliminates redundant information such as text content, including the table line feature information for constructing the table frame, denoted as the first table line feature information. Exemplarily: the number of horizontal table lines, the number of vertical tables, intersection features, boundary table line information; the cell information for constructing the table frame, denoted as the first cell feature information. Exemplarily: the number of cells, cell length, cell height, the number of horizontal cells, the number of vertical cells, adjacent cell association relationships, etc.

[0074] S30: Construct a first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate first cell positioning information, where the first cell positioning information corresponds one-to-one with the first cell set;

[0075] Specifically, the first positioning coordinate system refers to a virtual coordinate system for positioning cells constructed based on the first table line feature information, preferably a two-dimensional coordinate system. Taking an example without limitation for the construction process: a line parallel to the horizontal table line in the first table line feature information is used as the abscissa x-axis, and a line parallel to the vertical table line in the first table line feature information is used as the ordinate y-axis. Then, the unit length is determined according to the distance between the boundary table lines, and thus the intersection coordinates of all table lines are obtained.

[0076] The first cell positioning information refers to the cell positioning information generated by inputting the first cell feature information into the first positioning coordinate system and based on the intersection coordinates of the four table lines of the cell; the first cell set refers to all cell information in the first form information; the first cell positioning information and the first cell set are stored in a one-to-one correspondence for quick call in the subsequent steps.

[0077] S40: Traverse the first cell set based on the first cell positioning information to perform row and column vector identification, and generate a first cell row vector and a first cell column vector;

[0078] Specifically, the row vector and the column vector refer to two vectors perpendicular to each other in direction. In the first form information, the row vector is the vector information representing the horizontal information of the table and the table information recognition direction, and the column vector is the vector information representing the vertical information of the table and the table information recognition direction. In the coordinate system, the row vector is parallel to the abscissa x-axis, and the column vector is parallel to the ordinate y-axis.

[0079] The first cell row vector refers to the result after row vector identification of the first cell set by traversing the first cell positioning information; the first cell column vector refers to the result after column vector identification of the first cell set by traversing the first cell positioning information. By identifying the first cell row vector and the first cell column vector, it is convenient to divide the table into row information and column information for processing. Since the row information and column information in the table each have certain commonalities, the characteristics of the row information and column information can be quickly extracted by separate processing, and the redundancy is lower compared to the unified extraction in traditional technologies.

[0080] S50: Input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0081] Furthermore, based on inputting the first cell row vector and the first cell column vector into the row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information, step S50 further includes:

[0082] S501: Obtain a row vector feature extraction network and a column vector feature extraction network according to the row-column feature extraction network;

[0083] S502: Input the first cell row vector into the row vector feature extraction network to obtain first row feature information;

[0084] S503: Input the first cell column vector into the column vector feature extraction network to obtain first column vector feature information;

[0085] S504: Perform row-column feature fusion on the first row feature information and the first column vector feature information to obtain the first row-column feature information.

[0086] Specifically, the first row-column feature information refers to the feature information obtained by inputting the first cell row vector and the first cell column vector into the row-column feature extraction network for row-column feature extraction and then performing feature fusion on the row feature and the column feature.

[0087] It includes first semantic feature information representing text content, such as: semantic feature information such as subject feature, object feature, and attribute feature. For example: the subject feature is Li Si, the object features are: age, score, ranking, and the attribute feature is the specific value of the object feature.

[0088] Including first layout feature information characterizing the form of a table layout, for example: row-adjacent table layout, column-adjacent table layout. For example: row-adjacent feature information where a certain cell is aligned with one side of its adjacent row, and although the other side is boundary-aligned, it is horizontally divided into two cells.

[0089] The row vector feature extraction network refers to a functional module for extracting row vector features, constructed based on the attention mechanism and convolutional neural network. The attention mechanism is used in machine learning to focus on information in a certain dimension. By using the attention mechanism to focus on row vector information, row vector features are extracted in the row vector feature extraction network; the column vector feature extraction network refers to a functional module for extracting column vector features, constructed based on the attention mechanism and convolutional neural network. The attention mechanism is used in machine learning to focus on information in a certain dimension. By constructing the feature extraction network based on the convolutional neural network and the attention mechanism, row features and column features can be accurately extracted, providing a data basis for accurately determining the table type in the subsequent steps.

[0090] S60: Input the first semantic feature information and the first layout feature information into the table type recognition network for optimized screening of the table type, and obtain a first screening result;

[0091] Further, based on inputting the first semantic feature information and the first layout feature information into the table type recognition network for optimized screening of the table type and obtaining a first screening result, step S60 includes:

[0092] S601: Based on big data, perform table type matching according to the first layout feature information to obtain a first table type matching set;

[0093] S602: Construct a first optimized screening space according to the first table type matching set;

[0094] S603: Input the first semantic feature information into the first optimized screening space to obtain the first screening result.

[0095] Further, based on constructing a first optimized screening space according to the first table type matching set, step S602 includes:

[0096] S6021: Obtain a first matching degree calculation formula:

[0097]

[0098] where P n refers to the matching probability of the nth table type under the input semantic features, and f 1n represents the selection frequency of the nth table type in the local table type matching under the input semantic features, and f2n It represents the selection frequency of the nth table type under the input semantic features in the big data table type matching. α and β are custom proportional parameters characterizing f 1n and f 2n The weight, and N is the number of table types;

[0099] S6022: Construct the first optimized screening space according to the first matching degree calculation formula and the first table type matching set.

[0100] Furthermore, based on inputting the first semantic feature information into the first optimized screening space to obtain the first screening result, step S603 includes:

[0101] S6031: Obtain the first constraint function:

[0102] S6032: When P n+1 -P n ≥0, T = 1. When P n+1 -P n <0,

[0103] where T represents the acceptance degree of transferring from the nth table type to the n + 1th table type;

[0104] S6033: When P n+1 -P n <0, when T ≥ the first preset acceptance degree, accept the n + 1th table type.

[0105] Specifically, the first screening result refers to the data representing the table type matching result obtained by inputting the first semantic feature information and the first layout feature information into the table type recognition network for optimized screening of the table type. The table type recognition network is a functional module for table type screening constructed based on big data and optimization algorithms, and its working process is as follows:

[0106] The first table type matching set refers to the set of table types obtained by inputting the first layout feature information into big data for table type matching, that is, without considering semantic content, only performing table type matching in big data through the first layout feature information, and obtaining a finite set, which may include various existing known table types, including but not limited to table types with similar layouts, and table types with the same layout but different association relationships between cells; the first optimization screening space refers to a virtual function space constructed based on the limited first table type matching set for table type screening and optimization, where the first table type matching set is randomly assigned different coordinates and distributed in the first optimization screening space; the table types distributed in the first optimization screening space are all table types that match the first layout feature information, but the same table layout may record different semantic contents, so that even if the cell layouts are the same but the association relationships between cells are different, such different tables are also assigned different coordinates in the first optimization screening space. Further, inputting the first semantic feature information into the first optimization screening space to analyze the correlation between the table layout and the semantic features, and obtaining the final table type screening result after optimization screening, denoted as the first screening result.

[0107] The correlation evaluation method is preferably as follows:

[0108] According to the first matching degree calculation formula for evaluating the correlation between the evaluation table layout and the semantic features:

[0109]

[0110] Perform the evaluation, where P n refers to the matching probability of the nth table type under the input semantic features, and f 1n represents the selection frequency of the nth table type under the input semantic features in the local table type matching, and f 2n represents the selection frequency of the nth table type under the input semantic features in the big data table type matching. α and β are custom proportional parameters for characterizing the weights of f 1n and f 2n respectively, and N is the number of table types;

[0111] Perform table type screening depending on the first constraint function

[0112] When P n+1 - P n ≥0, T = 1; when P n+1 - P n <0,

[0113] where T represents the acceptance degree of transferring from the nth table type to the n + 1th table type;

[0114] At P n+1 -P n <0, when T ≥ the first preset acceptance degree, accept the (n + 1)-th table type.

[0115] Start traversing by randomly selecting a starting coordinate in the first optimization screening space, evaluate the relevance between the table layout and semantic features depending on the first matching degree calculation formula, and screen the transfer from the n-th table type to the (n + 1)-th table type depending on the difference value of the matching degree between the n-th table type and the (n + 1)-th table type according to the first constraint function. The first preset acceptance degree refers to when P n+1 -P n <0, the preset minimum acceptance degree, that is, when P n+1 -P n <0, P n+1 -P n The greater the difference between them, the smaller T is. Then, when it is greater than the first preset acceptance degree, if the difference between P n+1 -P n is not large, P does not need to be screened out n+1 , and P n+1 can be directly used as the traversing coordinate to traverse other table types. If T < the first preset acceptance degree, the table type corresponding to P n+1 will be eliminated.

[0116] Set a storage space to record the coordinate positions corresponding to the table types that have been traversed, including the table types retained by screening and the eliminated table types. When optimizing and traversing, directly skip the table types that have been traversed. Thus, global traversing is performed to generate the first screening result, and a screening result that fits the layout features and semantic features is obtained, without relying on expert experience for screening, improving the migration ability.

[0117] S70: Generate a first table type determination result according to the first screening result.

[0118] Specifically, the first table type determination result refers to the table type determination result determined by further selecting from the first screening result. After optimization screening, the number of table types is extremely small, and optionally, the staff can select from them. By default, the table types in the first screening result are sorted according to the matching degree from high to low, and the table type ranked first is output as the first table type determination result.

[0119] Furthermore, based on the row-column feature fusion of the first row feature information and the first column vector feature information to obtain the first row-column feature information, step S504 includes:

[0120] S5041a: Obtain a first linear layer and a second linear layer, where the first linear layer is used for semantic content matching, and the second linear layer is used for semantic content classification;

[0121] S5042a: Input the first row feature information into the first linear layer based on the attention mechanism to obtain a first semantic content matching result, where the first semantic content matching result includes first row adjacent semantic content;

[0122] S5043a: Input the first column feature information into the first linear layer based on the attention mechanism to obtain a second semantic content matching result, where the second semantic content matching result includes first column adjacent semantic content;

[0123] S5044a: Input the first row adjacent semantic content and the first column adjacent semantic content at the same coordinate position into the second linear layer to obtain first subject feature information, first object feature information, and first attribute feature information;

[0124] S5045a: Add the first subject feature information, the first object feature information, and the first attribute feature information into the first semantic feature information.

[0125] Specifically, the process of feature fusion is as follows:

[0126] The first linear layer is a fully connected neural network layer used for semantic content matching. The first semantic content matching result refers to the result obtained by inputting the first row feature information into the first linear layer for matching based on the attention mechanism. The second semantic content matching result refers to the result obtained by inputting the first column feature information into the first linear layer for matching based on the attention mechanism. The first linear layer can, based on the attention mechanism, extract the text features of the first column feature information and the first row feature information through the input first column feature information and first row feature information, while ignoring the frame features, and then match the corresponding semantic content information according to the text features. The first linear layer can divide it into corresponding content categories according to text features such as the character type, character length, and number of characters of the text, that is, realize semantic content matching.

[0127] Furthermore, the extracted semantic content still belongs to the corresponding cell coordinates in the first coordinate system. The first row adjacent semantic content refers to the semantic content information in the row adjacent cells, and the first column adjacent semantic content refers to the semantic content information in the column adjacent cells.

[0128] The second linear layer is a fully-connected layer of a neural network for semantic content classification, that is, a fully-connected network for analyzing the correlation relationships between the semantic contents in adjacent cells. In order to fuse the adjacent semantic contents in the first row and the adjacent semantic contents in the first column, the second linear layer traverses the adjacent semantic contents in the first row and the adjacent semantic contents in the first column of the cells at the same coordinate position based on the cell positioning information, so as to determine the correlation relationship between the current cell and its adjacent semantic contents in the row, determine the correlation relationship between the current cell and its adjacent semantic contents in the column, and further determine the correlation relationship between its adjacent semantic contents in the row and its adjacent semantic contents in the column. Generally speaking, it is divided into: the first subject feature information, the first object feature information, and the first attribute feature information, which are illustrated with examples above. Add the first subject feature information, the first object feature information, and the first attribute feature information into the first semantic feature information. By fusing the semantic features, the semantic content and its correlation relationship are obtained, providing an important data basis for subsequent table recognition.

[0129] Further, based on the row-column feature fusion of the first row feature information and the first column vector feature information to obtain the first row-column feature information, step S504 includes:

[0130] S5041b: Obtain a third linear layer and a fourth linear layer, where the third linear layer is used for layout form matching, and the fourth linear layer is used for layout form classification;

[0131] S5042b: Input the first row feature information into the third linear layer based on the attention mechanism to obtain a first layout form matching result, where the first layout form matching result includes the layout form information of adjacent cells in the first row;

[0132] S5043b: Input the first column feature information into the third linear layer based on the attention mechanism to obtain a second layout form matching result, where the second layout form matching result includes the layout form information of adjacent cells in the first column;

[0133] S5044b: Input the layout form information of adjacent cells in the first row and the layout form information of adjacent cells in the first column at the same coordinate position into the fourth linear layer to generate first layout form classification information;

[0134] S5045b: Add the first layout form classification information into the first layout feature information.

[0135] Specifically, the process of feature fusion is as follows:

[0136] The third linear layer is a fully-connected layer of a neural network used for layout form matching. The first layout form matching result refers to the result obtained by inputting the first row feature information into the first linear layer for matching based on the attention mechanism. The second layout form matching result refers to the result obtained by inputting the first column feature information into the first linear layer for matching based on the attention mechanism. The third linear layer can, through the input first column feature information and first row feature information, extract the frame features of the first column feature information and the first row feature information respectively based on the attention mechanism, while ignoring the text features, and then match the corresponding cell layout form information according to the frame features. The third linear layer can divide it into corresponding table form categories according to table forms, such as the cell length of the table, the number of cells, the number of table lines, etc., that is, to achieve layout form matching.

[0137] Furthermore, the extracted layout form still belongs to the corresponding cell coordinates in the first coordinate system. The first row adjacent cell layout form information refers to the cell form information within the row adjacent cells. The first column adjacent cell layout form information refers to the cell form information within the column adjacent cells.

[0138] The fourth linear layer is a fully-connected layer of a neural network used for layout form classification, that is, a fully-connected network used to analyze the association relationship between the cell forms within adjacent cells. In order to fuse the first row adjacent cell layout form information and the first column adjacent cell layout form information, the fourth linear layer traverses the first row adjacent cell layout form information and the first column adjacent cell layout form information of the cells at the same coordinate position based on the cell positioning information, and then determines the association relationship between the current cell and its row adjacent cell layout form, and determines the association relationship between the current cell and its column adjacent cell layout form. The cell form matching data depends on big data collection and matching, and the first layout form classification information is added to the first layout feature information. By fusing the layout features, the cell layout form and the association relationship between cells are obtained, providing an important data basis for subsequent table recognition.

[0139] Refer to Figure 2 As shown, it is a functional network schematic diagram of the neural network model 100 of the present application.

[0140] The neural network model 100 described in the present application can be stored in an electronic device. According to the implemented functions, the neural network model 100 can include a correction network 110, a frame feature extraction network 120, a row and column feature extraction network 130, and a table type recognition network 140. This network refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions.

[0141] In this embodiment, asFigure 3 As shown in the figure, it is a table type determination system based on a neural network model provided by an embodiment of the present application. The neural network model includes a correction network, a frame feature extraction network, a row-column feature extraction network, and a table type recognition network. The system includes:

[0142] A first acquisition unit 21, configured to input first form information into the correction network to obtain first corrected result information;

[0143] A second acquisition unit 22, configured to input the first corrected result information into the frame feature extraction network to obtain first table frame feature information, where the first table frame feature information includes first table line feature information and first cell feature information;

[0144] A first generation unit 23, configured to construct a first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate first cell positioning information, where the first cell positioning information corresponds one-to-one with a first cell set;

[0145] A second generation unit 24, configured to perform row-column vector identification on the first cell set based on the first cell positioning information, and generate a first cell row vector and a first cell column vector;

[0146] A third acquisition unit 25, configured to input the first cell row vector and the first cell column vector into the row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0147] A fourth acquisition unit 26, configured to input the first semantic feature information and the first layout feature information into the table type recognition network for table type optimization and screening to obtain a first screening result;

[0148] A third generation unit 27, configured to generate a first table type determination result according to the first screening result.

[0149] Refer to Figure 4 As shown in the figure, it is a schematic diagram of a preferred embodiment of the electronic device 1 of the present application.

[0150] The electronic device 1 includes, but is not limited to: a memory 11, a processor 12, a display 13, and a network interface 14. The electronic device 1 connects to a network through the network interface 14 to obtain original data. Among them, the network can be a wireless or wired network such as an enterprise intranet (Intranet), the Internet, Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, a call network, etc.

[0151] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as the hard disk or memory of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped with the electronic device 1, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Of course, the memory 11 may also include both the internal storage unit and the external storage device of the electronic device 1. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed in the electronic device 1, such as the program code of the table type determination program 10 based on a neural network model, etc. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or will be output.

[0152] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication, etc. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the program code of the table type determination program 10 based on a neural network model, etc.

[0153] The display 13 may be referred to as a display screen or a display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch liquid crystal display, an organic light-emitting diode (OLED) touch device, etc. The display 13 is used to display the information processed in the electronic device 1 and to display a visual working interface, such as the result of data statistics display.

[0154] The network interface 14 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and this network interface 14 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0155] Figure 3 Only the electronic device 1 with components 11-14 and the table type determination program 10 based on the neural network model is shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0156] Optionally, the electronic device 1 may further include a user interface. The user interface may include a display, an input unit such as a keyboard, and the optional user interface may also include a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, an organic light-emitting diode (OLED) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, and is used to display the information processed in the electronic device 1 and to display a visual user interface.

[0157] The electronic device 1 may further include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be elaborated here.

[0158] In the above embodiments, when the processor 12 executes the table type determination program 10 stored in the memory 11, the following steps may be implemented:

[0159] Input the first form information into the correction network to obtain the first corrected result information;

[0160] Input the first corrected result information into the frame feature extraction network to obtain the first table frame feature information, and the first table frame feature information includes the first table line feature information and the first cell feature information;

[0161] Construct a first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate first cell positioning information;

[0162] Traverse the first cell set based on the first cell positioning information for row and column vector identification, and generate a first cell row vector and a first cell column vector;

[0163] Input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0164] Input the first semantic feature information and the first layout feature information into a table type recognition network for table type optimization and screening to obtain a first screening result;

[0165] Generate a first table type determination result according to the first screening result.

[0166] The storage device can be the memory 11 of the electronic device 1 or other storage devices communicatively connected to the electronic device 1.

[0167] For a detailed introduction to the above steps, please refer to the above Figure 1 Description of the flowchart of the method embodiment for determining the table type based on the neural network model.

[0168] In addition, an embodiment of the present application also proposes a computer-readable storage medium, which can be non-volatile or volatile. The computer-readable storage medium can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium includes a storage data area and a storage program area. The storage data area stores data created according to the use of the blockchain node, and the storage program area stores a table type determination program 10 based on the neural network model. When the table type determination program 10 based on the neural network model is executed by a processor, the following operations are implemented:

[0169] Input the first form information into a correction network to obtain first correction result information;

[0170] Input the first correction result information into a frame feature extraction network to obtain first table frame feature information, where the first table frame feature information includes first table line feature information and first cell feature information;

[0171] Construct a first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate first cell positioning information;

[0172] Traverse the first cell set based on the first cell positioning information for row and column vector identification, and generate a first cell row vector and a first cell column vector;

[0173] Input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information;

[0174] Input the first semantic feature information and the first layout feature information into a table type recognition network for table type optimization and screening to obtain a first screening result;

[0175] Generate a first table type determination result according to the first screening result.

[0176] The specific implementation manner of the computer-readable storage medium of the present application is substantially the same as the specific implementation manner of the above table type determination method based on a neural network model, and will not be elaborated here.

[0177] This application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) uses a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results in theory, methods, technologies, and application systems.

[0178] Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0179] In another embodiment, for the table type determination method based on a neural network model provided by the present application, to further ensure the privacy and security of all the data appearing above, all the above data can also be stored in a node of a blockchain. For example, table types and table feature information, etc., these data can all be stored in the blockchain node.

[0180] It should be noted that the blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithms. Blockchain, in essence, is a decentralized database, a series of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of the information (anti-counterfeiting) and generate the next block.

[0181] It should be noted that the serial numbers of the embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments. And the term "including" or "comprising" or any other variant thereof in this article is intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, device, article or method. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, device, article or method including the element.

[0182] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium as described above (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, electronic device, or network device, etc.) to execute the methods described in various embodiments of the present application.

Claims

1. A method for determining table types based on a neural network model, characterized in that, The neural network model includes a correction network, a frame feature extraction network, a row-column feature extraction network, and a table type recognition network. The method includes: Input the first form information into the correction network to obtain the first correction result information; Input the first correction result information into the frame feature extraction network to obtain the first table frame feature information, where the first table frame feature information includes the first table line feature information and the first cell feature information; Construct a first positioning coordinate system according to the first table line feature information, and input the first cell feature information into the first positioning coordinate system to generate the first cell positioning information. Among them, the first cell positioning information corresponds one-to-one with the first cell set, and the first cell set includes all cell information in the first form information; Traverse the first cell set based on the first cell positioning information for row-column vector identification to generate the first cell row vector and the first cell column vector; Input the first cell row vector and the first cell column vector into the row-column feature extraction network to obtain the first row-column feature information, where the first row-column feature information includes the first semantic feature information and the first layout feature information; Based on big data, perform table type matching according to the first layout feature information to obtain the first table type matching set, construct the first optimization and screening space according to the first table type matching set, and input the first semantic feature information into the first optimization and screening space to obtain the first screening result; Generate the first table type determination result according to the first screening result; The constructing the first optimization and screening space according to the first table type matching set includes: Obtain the first matching degree calculation formula; Among them, refers to the matching probability of the nth table type under the input semantic features, represents the selection frequency of the nth table type under the input semantic features in the local table type matching, represents the selection frequency of the nth table type under the input semantic features in the big data table type matching, is used to represent and the custom proportional parameter of the weight, is the number of table types; Construct the first optimization and screening space according to the first matching degree calculation formula and the first table type matching set; The inputting the first semantic feature information into the first optimization and screening space to obtain the first screening result includes: Obtain the first constraint function; When then When then Among them, indicates the acceptance degree transferred from the table type to the table type. At When ≥ the first preset acceptance level, accept the table type.

2. The method according to claim 1, characterized in that, The inputting the first cell row vector and the first cell column vector into the row-column feature extraction network to obtain the first row-column feature information includes: Obtain the row vector feature extraction network and the column vector feature extraction network according to the row-column feature extraction network; Input the first cell row vector into the row vector feature extraction network to obtain the first row feature information; Input the first cell column vector into the column vector feature extraction network to obtain the first column feature information; Perform row-column feature fusion on the first row feature information and the first column feature information to obtain the first row-column feature information.

3. The method according to claim 2, characterized in that, The performing row-column feature fusion on the first row feature information and the first column feature information to obtain the first row-column feature information includes: Obtain the first linear layer and the second linear layer, where the first linear layer is used for semantic content matching, and the second linear layer is used for semantic content classification; Input the first row feature information into the first linear layer based on the attention mechanism to obtain the first semantic content matching result, where the first semantic content matching result includes the first adjacent row semantic content; Input the first column feature information into the first linear layer based on the attention mechanism to obtain a second semantic content matching result, where the second semantic content matching result includes adjacent semantic content in the first column; Input the adjacent semantic content in the first row and the adjacent semantic content in the first column at the same coordinate position into the second linear layer to obtain first subject feature information, first object feature information, and first attribute feature information; Add the first subject feature information, the first object feature information, and the first attribute feature information to the first semantic feature information.

4. The method according to claim 2, characterized in that, The performing row-column feature fusion on the first row feature information and the first column feature information to obtain the first row-column feature information includes: Obtain a third linear layer and a fourth linear layer, where the third linear layer is used for layout form matching, and the fourth linear layer is used for layout form classification; Input the first row feature information into the third linear layer based on the attention mechanism to obtain a first layout form matching result, where the first layout form matching result includes adjacent cell layout form information in the first row; Input the first column feature information into the third linear layer based on the attention mechanism to obtain a second layout form matching result, where the second layout form matching result includes adjacent cell layout form information in the first column; Input the adjacent cell layout form information in the first row and the adjacent cell layout form information in the first column at the same coordinate position into the fourth linear layer to generate first layout form classification information; Add the first layout form classification information to the first layout feature information.

5. A table type determination device based on a neural network model, which is used to implement the table type determination method based on the neural network model according to any one of claims 1 to 4, and is characterized in that, The device includes: A first obtaining unit, configured to input first form information into a correction network to obtain first correction result information; A second obtaining unit, configured to input the first correction result information into a frame feature extraction network to obtain first table frame feature information, where the first table frame feature information includes first table line feature information and first cell feature information; A first generating unit, configured to construct a first positioning coordinate system according to the first table line feature information, input the first cell feature information into the first positioning coordinate system, and generate first cell positioning information, where the first cell positioning information corresponds one-to-one with a first cell set, and the first cell set includes all cell information in the first form information; A second generating unit, configured to perform row-column vector identification on the first cell set by traversing the first cell positioning information to generate a first cell row vector and a first cell column vector; A third obtaining unit, configured to input the first cell row vector and the first cell column vector into a row-column feature extraction network to obtain first row-column feature information, where the first row-column feature information includes first semantic feature information and first layout feature information; A fourth obtaining unit, configured to input the first semantic feature information and the first layout feature information into a table type recognition network for table type optimization and screening to obtain a first screening result; A third generating unit, configured to generate a first table type determination result according to the first screening result.

6. An electronic device, characterized in that, The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, the memory stores a program executable by the at least one processor, and when the program is executed by the at least one processor, the at least one processor is enabled to execute the table type determination method based on a neural network model according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a table type determination program based on a neural network model. When the table type determination program based on the neural network model is executed by a processor, the steps of the table type determination method based on the neural network model according to any one of claims 1 to 4 are implemented.

Images