An Optimized System for Mapping Non-Structured Financial Excel Tables to Databases

By combining rules and machine learning algorithm models, using neural networks and DBSCAN clustering algorithms to optimize the coding layer, the standardization and structure of Excel tables are achieved, and the problem of low accuracy of table structure recognition and data mapping in the existing technology is solved. Achieve an accuracy rate of 96%-98% to meet financial business needs.

CN113761202BActive Publication Date: 2024-11-05BEIJING KUAQUO INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111000985.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-30
Publication Date
2024-11-05
Estimated Expiration
2041-08-30

AI Technical Summary

Technical Problem

When the existing technology converts Excel table data into structured data, there are problems such as numerous manual rule logic and conflicting. Deep learning neural network methods lose clear rule information in image processing, resulting in the inability to identify table structures and data mapping accuracy. It is difficult to achieve the commercial level, especially in the common domain of the financial field.

Method used

Using a system based on rules and machine learning algorithm models, the mapping relationship between table column names and database column names is predicted through neural network models, combined with DBSCAN clustering algorithm and optimized coding layer, the standardization and structure of Excel tables are realized, and the segmentation is gradually split. Merge cells to form a standard M*N table and map the table data to the database.

Benefits of technology

It has achieved standardization of non-structural information in Excel tables, achieving an accuracy rate of 96%-98%, meeting the commercial requirements of financial business, and improving the accuracy of table structure identification and data mapping.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113761202B_ABST
    Figure CN113761202B_ABST
Patent Text Reader

Abstract

An optimization system for mapping non-structural financial Excel tables to databases, which relates to the field of data processing technology. It includes the following specific processes: providing a standard Excel file input service interface to receive Excel table data; judging how many sheet files exist in Excel, and performing the following operations on each sheet file: operating each sheet in turn; operating a single sheet file in turn; building a set of neural network models to integrate the column name and content of each column, and predicting the mapping relationship between each column information and the column name in the database through the neural network model; and writing the results to a structured database. The beneficial effects of the present invention are: realizing the standardization of non-structural information in Excel tables, and mapping it to the corresponding columns of the database based on the data in the table, achieving an accuracy rate of 96%-98% in financial business, which basically meets commercial requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an optimization system for mapping a non-structural financial Excel table to a database. Background Art

[0002] In data analysis and processing, it is usually necessary to organize complex table data into a standard data format. Complex tables usually include various forms such as "merging cells, splitting cells" in the table structure, and "changing the background color of the table, the color of the text, and bolding the text" to express key data, table headers and other information in the table. For this type of data, this patent aims to solve the problem of determining the area of ​​the financial table, the table header, column name, and data in the table through an algorithm model, and performing reverse operations for the "merging and splitting cells" situation to restore the table.

[0003] There are many existing technologies for converting Excel table data into structured table data. However, they can be divided into two technical directions. One is the rule indexing method, and the second is the deep learning neural network technology developed in recent years. Rule indexing mainly uses certain rules and procedures to read the data and format (such as whether to merge, font size, bold, background color, etc.) in the Excel table, and then manually organizes a set of rule logic to judge the table area, header, column name and specific data value and other information. The deep neural network technology developed in recent years currently mainly solves the problem of table area recognition and positioning in the "image", as well as the relationship between cells in the table, that is, whether to merge, split, or whether they are in the same row or column. However, the following shortcomings still exist:

[0004] With existing technologies, there are usually two solutions for table structure recognition. The first is to judge by rule index. Its disadvantage is that it requires a lot of manual work to sort out various rule logics. Due to the large number of rules, it cannot cover all situations, and it is often easy to have conflicts between rules. The second is a neural network-based method, which has certain learning and generalization capabilities for tables in images. However, after converting Excel data into images, some clear rule information and information on numerical values ​​in the table are lost; this results in the accuracy rate not yet reaching commercial levels.

[0005] Secondly, for table structure recognition and data mapping in general domains, usually only unstructured tables are converted into standard tables. It is not possible to do associative mapping of data in the table, that is, how to judge the corresponding table header through the specific values ​​in the table. Taking the time "2021-12-31" as an example, in the field of financial bonds, this value may be "issuance date" or "deadline", "tender start date", "tender end date", etc. If it is placed in the general domain, it has more possibilities, so it is difficult to implement in the general domain, according to the value in the table, directly judge whether it corresponds to the column in the database. Summary of the invention

[0006] The purpose of the present invention is to provide an optimization system for mapping non-structured financial Excel tables to databases in view of the deficiencies and shortcomings in the prior art, thereby realizing the standardization of non-structured information in Excel tables and mapping them to corresponding columns in the database based on the data in the tables, achieving an accuracy rate of 96%-98% in financial business, which basically meets commercial requirements.

[0007] To achieve the above object, the present invention adopts the following technical solution: an optimization system for mapping non-structured financial Excel tables to a database, which includes the following specific processes: providing a standard Excel file input service interface to receive Excel table data;

[0008] Determine how many sheet files exist in Excel, and perform the following operations on each sheet file: operate on each sheet in turn;

[0009] The following operations are performed on a single sheet file: first, the status information of the sheet is determined and recorded in the status library; second, based on the rules and machine learning algorithm model, the number of independent tables, the title of each table, and the header information are determined; finally, the table is reorganized, and the merged cells are split to form a standard M*N table;

[0010] Build a neural network model to integrate the column name and content of each column, and use the neural network model to predict the mapping relationship between each column information and the column name in the database;

[0011] Write the results to the structure database.

[0012] Furthermore, a standard Excel file input service interface is provided, and the service interface for receiving Excel table data includes the following steps: providing a standard web service interface, receiving the Excel file sent by the front end, and performing file format judgment, sheet format judgment in the file, recording the number of sheets, and extracting data in the sheet table one by one; for each sheet table, calling the "table structure recognition" model and the "table data mapping" model one by one, obtaining the standardized table data in each sheet, and arranging it into a standard format; the service interface returns the standardized data, writes it into the structure database or returns it to the front end for display, and completes the complete process of Excel table mapping.

[0013] Furthermore, the extraction of table status information from several sheet files in Excel includes the following steps: status recorder: recording the start row / column, the end row / column; which rows are merged cells, which rows are bold text rows, and which rows have the background color of the entire row modified; using rules and clustering algorithms to disassemble sub-tables; identifying table element titles, column names, and data; standardizing tables, recombining sub-tables, and splitting merged cells.

[0014] Furthermore, the status recorder includes the following specific processes: reading the table information of a sheet through an open source plug-in, including the number of rows and columns; and the starting row and the ending row. The specific method is to determine whether it is the starting row or column by judging whether there is data in each row and each column; logically judging whether each row has merged cells, whether the entire row has a background color, and whether the entire row has a bold font, and writing this information into the status recorder respectively; calculating the length of each row, taking into account that some rows are subsequently empty, or there are multiple tables in a sheet; calculating the information of the ending column of each row, and the rule of the end point is that there is no data in the cells of the current row.

[0015] Furthermore, the sub-table decomposition using rules and clustering algorithms includes the following operation flow: using rules to decompose a table in a sheet, the rules include the following points: Rule 1: the situation where the whole row of cells is merged is defined as the segmentation standard between multiple sub-tables, Rule 2: the situation where the whole row of cell data is bolded is defined as the segmentation standard between multiple sub-tables, Rule 3: the whole row of table data is added with a background color, and the background color of the previous row is inconsistent with the background color of the current row, which is defined as the segmentation standard between multiple sub-tables, Rule 4: using a clustering algorithm, clustering is performed according to the length value of each row and the feature characteristics of the elements in the row, and according to the clustering result, when two adjacent rows are not in the same category, it is defined as the segmentation standard of multiple sub-tables themselves, and the specific implementation method of clustering: based on the DBSCAN clustering algorithm, combined with the text feature input embedding information, the centroid is made to wander up and down multiple times to record a certain length coding distribution offset (Euclidean distance + data type KLdistance), if there is an obvious offset and it is close to the offset of the current list header, it is judged that the upper and lower are table boundaries.

[0016] Furthermore, the identification of the table element title, column name, and data includes the following identification methods: table title identification, using rule logic, rule: when merging cells in an entire row, the cells are merged downward by default, and serve as the title or note information of the current table; table column name identification, using rule logic, rule 1: if the entire row is in bold font, it is the column name of the table by default; rule 2: if the entire row has a background color, and the previous row and the next row have no background color or the background color is different from the current row, it is the column name of the table by default; table data identification, data that does not meet the above title and column name logic rules is defaulted to being data information in the table.

[0017] Furthermore, the table standardization, sub-table regrouping and merged cell splitting include the following operation flow: sub-table regrouping, for each sub-table, rule 1: if the column name of the current table is the same as the column name of the next sub-table, or one of the sub-table column names completely belongs to the other sub-table column name, the two sub-tables are merged; for each sub-table, according to the table status, if it is determined that there are merged cells, the merged cells are disassembled, and the disassembled cells are assigned the same value, thereby forming one or more standard M*N cells.

[0018] Furthermore, the operation on a single sheet file is performed based on how many sheet files exist in Excel, and we form one or more standard M*N sub-tables; each incomplete table contains four parts of information: table title, header, table data, and remarks, among which the table title and remarks are the text information of the entire row of merged cells; the header and table data are standard M*N data, and this solution needs to map them one by one with the column names in the database based on their column names and table data, and the table data mapping includes the following processes: data preparation, considering that a column of data in each table may be very long, the solution adopts a random extraction method to extract 5-10 cell data and column names from a column of data, and combine them into a group of column information. Each column of data uses the same method to form N groups of corresponding prediction data, and construct a deep learning neural network, whose network structure is shown in the fourth step of the solution. Through this algorithm model, we predict each column of data and determine its mapping relationship with the column information in the database. For each sub-table, the final complete table structure information is output, including the table title, column name, data, remarks information, and the corresponding relationship between the information of each column in the database.

[0019] Furthermore, the construction of a neural network model integrates the column name and content of each column, and predicts the mapping relationship between each column information and the column name in the database through the neural network model. The table data classification model includes the following encoding layers and processes: encoding layer, considering that each table may contain long text, string, floating point number and integer data types, and floating point, integer and string information are crucial in the financial system, it is inappropriate to use the native pre-trained model to pre-process floating point numbers and integers, so the entire pre-trained encoding layer is optimized; numerical encoding layer: the location of the numerical value, the numerical value is encoded separately: according to the characteristics of the corpus, The upper limit of 5 data is supported to enter the numerical coding system. Each bit of each value, including the positive and negative signs, supports 8 bits for the integer part and 4 bits for the decimal part. Each digit is losslessly one-hot encoded, which is parallel to the text / string encoding. The text part takes the sequence mean, the highest value, and the end sequence for merging; the dimension reduction layer, due to the large dimension of the encoding layer, adopts a similar text-cnn method to reduce the dimension here. 1d uses a low out-channel and a relatively large stride, and cooperates with a linear dimensionality reduction autoencoder to perform residual splicing to achieve dimensionality reduction operations; the output layer uses a fully connected layer to access the upper feature map. Due to the dimensional changes that may be caused by the upper layer splicing operation, batch-regularization is used, and 10% of the parameters are dropped to prevent overfitting, and then the above-mentioned fully connected layer is connected for calculation. The loss adopts multi-classification cross entropy, and the weights of the imbalanced part are adjusted; the operation is performed using a neural network model structure; the problem of mapping each column of data to the corresponding column of the database.

[0020] Furthermore, the optimization content of the encoding layer includes the following: optimizing the pre-training model to solve the problem that floating-point numbers in the native pre-training model will be forcibly separated by decimal points to form separate tokens; optimizing the pre-training encoding layer. In the native pre-training model, floating-point numbers and integers are often non-financial backgrounds in Wikipedia or other Chinese and English pre-training corpora, and there is no accuracy requirement, and they can be replaced arbitrarily, such as years and ordinary numbers. Even if they are replaced, the order change will not affect the pre-training model. The financial context requires very precise values, and the value range itself has meaning and cannot be easily replaced or reordered. This solution solves this problem; solves the problem that the numerical part and the string in the native pre-training model are treated equally, the training amount is obviously insufficient, and it is easily affected by the forgetting characteristics of the neural network; the compression rate of the native pre-training model is high, and the numbers are saved to 312 dimensions, which loses the size order information and shares the channel with a large number of other characters, resulting in serious data loss. This solution optimizes this problem.

[0021] After adopting the above technical solution, the beneficial effects of the present invention are: standardization of non-structured information in Excel tables is achieved, and it is mapped to the corresponding columns of the database based on the data in the table, achieving an accuracy rate of 96%-98% in financial business, basically meeting commercial requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 It is the optimization system flow chart of the present invention.

[0024] Figure 2 It is a complete neural network model structure diagram in the present invention. DETAILED DESCRIPTION

[0025] See also Figure 1-Figure 2 As shown, the technical solution adopted by this specific implementation is:, it includes the following specific processes: providing a standard Excel file input service interface to receive Excel table data;

[0026] Determine how many sheet files exist in Excel, and perform the following operations on each sheet file: operate on each sheet in turn;

[0027] The following operations are performed on a single sheet file: first, the status information of the sheet is determined and recorded in the status library; second, based on the rules and machine learning algorithm model, the number of independent tables, the title of each table, and the header information are determined; finally, the table is reorganized, and the merged cells are split to form a standard M*N table;

[0028] Build a neural network model to integrate the column name and content of each column, and use the neural network model to predict the mapping relationship between each column information and the column name in the database;

[0029] Write the results to the structure database.

[0030] The described method provides a standard Excel file input service interface, and the service interface for receiving Excel table data includes the following steps: providing a standard web service interface, receiving the Excel file sent by the front end, and performing file format judgment, sheet format judgment in the file, recording the number of sheets, and extracting data in the sheet tables one by one; for each sheet table, calling the "table structure recognition" model and the "table data mapping" model one by one, obtaining the standardized table data in each sheet, and arranging it into a standard format; the service interface returns the standardized data, writes it into the structure database or returns it to the front end for display, and completes the complete process of Excel table mapping.

[0031] The method for extracting table status information from several sheet files in Excel includes the following steps: status recorder: recording the starting row / column and the ending row / column; which rows are merged cells, which rows are bold text rows, and which rows are all background colors of the entire row are modified; using rules and clustering algorithms to disassemble sub-tables; identifying table element titles, column names, and data; standardizing tables, recombining sub-tables, and splitting merged cells. The status recorder includes the following specific processes: reading the table information of a sheet through an open source plug-in, including the number of rows and columns; and the starting row and ending row. The specific method is to determine whether a row or column is started by determining whether there is data in each row and each column; logically determine whether each row is a merged cell row, whether the entire row has a background color, and whether the entire row has a bold font, and write this information into the status recorder respectively; calculating the length of each row, taking into account the problem that some rows are subsequently empty, or there are multiple tables in a sheet; calculating the information of the ending column of each row, and ending The rule of the point is that there is no subsequent data in the cells of the current row. The use of rules and clustering algorithms to disassemble subtables includes the following operation process: Use rules to disassemble a table in a sheet. The rules include the following points: Rule 1: When the entire row of cells is merged, it is defined as the segmentation standard between multiple subtables. Rule 2: When the data of the entire row of cells is bold, it is defined as the segmentation standard between multiple subtables. Rule 3: The entire row of table data is added with a background color, and the background color of the previous row is inconsistent with the background color of the current row, which is defined as the segmentation standard between multiple subtables. Rule 4: Use a clustering algorithm to cluster according to the length value of each row and the feature characteristics of the row. According to the clustering results, when two adjacent rows are not in the same category, it is defined as the segmentation standard of multiple subtables. The specific implementation method of clustering: Based on the DBSCAN clustering algorithm, the text feature input embedding information is combined to make the centroid walk up and down multiple times to record a certain length encoding distribution offset (Euclidean distance + data type KL distance), if there is an obvious offset and it is close to the offset of the current list header, then the upper and lower parts are judged to be table boundaries. The identification of the table element title, column name, and data includes the following identification methods: table title identification, using rule logic, rule: the whole row merges cells, and the default merge is downward, and it is used as the title or remark information of the current table; table column name identification, using rule logic, rule 1: if the whole row font is bold, it is the table column name by default; rule 2: if the whole row has a background color, and the previous row and the next row have no background color or the background color is different from the current row, it is the table column name by default;Table data identification, data that does not meet the above title and column name logic rules, defaults to data information in the table, the table standardization, sub-table re-combination and merged cell splitting include the following operation flow: Sub-table re-combination, for each sub-table, rule 1: if the column name of the current table is the same as the column name of the next sub-table, or one sub-table column name completely belongs to the other sub-table column name, then the two sub-tables are merged; for each sub-table, according to the table status, if there is a merged cell, the merged cell is disassembled, and the disassembled cells are assigned the same value, thereby forming one or more standard M*N cells. ;

[0032] The operation on a single sheet file is performed based on how many sheet files exist in Excel. We form one or more standard M*N sub-tables; each incomplete table contains four parts of information: table title, header, table data, and remarks, among which the table title and remarks are the text information of the entire row of merged cells; the header and table data are standard M*N data. This solution needs to map them one by one with the column names in the database based on their column names and table data, and the table data mapping includes the following processes: data preparation. Considering that a column of data in each table may be very long, the solution adopts a random extraction method to extract 5-10 cell data and column names from a column of data, and combine them into a group of column information. Each column of data uses the same method to form N groups of corresponding prediction data, and construct a deep learning neural network. Its network structure is shown in the fourth step of the solution. Through this algorithm model, we predict each column of data and determine its mapping relationship with the column information in the database. For each sub-table, the final complete table structure information is output, including the table title, column name, data, remarks information, and the corresponding relationship between the information of each column in the database.

[0033] The above-mentioned neural network model is constructed to integrate the column name and content of each column. Through the neural network model, the mapping relationship between each column information and the column name in the database is predicted. The table data classification model includes the following encoding layers and processes: encoding layer, considering that each table may contain long text, string, floating point number and integer data types, and floating point, integer and string information are crucial in the financial system, it is inappropriate to use the native pre-trained model to pre-process floating point numbers and integers, so the entire pre-trained encoding layer is optimized; numerical encoding layer: the location of the numerical value is separately encoded. Method: according to the characteristics of the corpus, the upper limit of 5 data is supported Entering the numerical encoding system, each bit of each value, including the positive and negative signs, supports 8 bits for the integer part and 4 bits for the decimal part. Each digit is losslessly one-hot encoded, which is parallel to the text / string encoding. The text part takes the sequence mean, the highest value, and the end sequence for merging; in the dimensionality reduction layer, since the dimension of the encoding layer is large, a similar method to text-CNN is used for dimensionality reduction. 1D uses a low out-channel and a relatively large stride, and cooperates with a linear dimensionality reduction autoencoder to perform residual splicing to achieve dimensionality reduction operations; the output layer uses a fully connected layer to access the upper feature map. Due to the splicing operation of the upper layer, The dimensional changes that can be caused are treated with batch-regularization, and 10% of the parameters are dropped to prevent overfitting. Then the above-mentioned fully connected layer is connected for calculation, and the multi-classification cross entropy loss is adopted, and the weight of the imbalanced part is adjusted; the neural network model structure is used for operation; each column of data is mapped to the corresponding column of the database. The optimization content of the encoding layer includes the following: optimizing the pre-training model to solve the problem that floating-point numbers in the native pre-training model will be forced to be separated by decimal points to form separate tokens; optimizing the pre-training encoding layer, the floating-point numbers and integers in the native pre-training model are often non- Financial background, no precision requirements, can be replaced arbitrarily, such as years and ordinary values. Even if replaced, the reorder will not affect the pre-trained model. The financial context requires very precise values, and the value range itself has meaning, which cannot be easily replaced or reordered. This solution solves this problem; it solves the problem that the numerical part and the string in the native pre-trained model are treated the same, the training amount is obviously insufficient, and it is easily affected by the forgetting characteristics of the neural network; the compression rate of the native pre-trained model is high, the numbers are saved to 312 dimensions, the size order information is lost and the channel is shared with a large number of other characters, resulting in serious data loss. This solution optimizes this problem.

[0034] Note: The extended technical features and application scenarios of this invention are as follows:

[0035] 1. Use the DBSCAN clustering algorithm to integrate the Excel table status information extracted by rules to distinguish and identify the structural information of the Excel table, including "table title, column name, data" and "multiple sub-tables". This rule index + machine learning method can be expanded or applied to other table recognition fields.

[0036] 2. The data encoding mode containing "large numbers" in financial tables can be used as an independent module or processing method and applied to the encoding layer or pre-training layer of related deep learning containing "large numbers" in other fields.

[0037] 3. This model encoding method that integrates "rule labels" can also be extended to other related tasks that need to be combined with "external rules".

[0038] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. An optimization method for mapping non-structured financial Excel tables to a database, characterized by: It includes the following specific processes: Step 1: Provide a standard Excel file input service interface to receive Excel table data; Step 2: Determine how many sheet files exist in Excel, perform the following operations on each sheet file, and perform step 3 on each sheet in turn; Step 3: Perform the following operations on a single sheet file: a) First determine the status information of the sheet and record it in the status library; b) Secondly, based on the rules and machine learning algorithm model, determine how many independent tables there are, the title of each table, and the header information; c) Finally, reorganize the table and split the merged cells to form a standard M*N table; Step 4: Build a neural network model to integrate the column name and content of each column. Use the neural network model to predict the mapping relationship between each column information and the column name in the database. Step 5: Write the results into the structure database; The service interface in step 1 includes the following types: 1.1) Provide a standard web service interface to receive the Excel file sent by the front end, determine the sheet format in the file, record the number of sheets, and extract the data in the sheet one by one; 1.2) For each sheet, call the table structure recognition model and table data mapping model one by one to obtain the standardized table data in each sheet and organize it into a standard format; 1.3) Service interface, returns the standardized data, writes it into the structure database or returns it to the front-end for display, completing the complete process of Excel table mapping; The extraction of table status information includes the following steps: Step 2.1: Status recorder: record the start row / column, end row / column, which rows are merged cells, which rows are bold, and which rows are background colors of the entire row; Step 2.2: Use rules and clustering algorithms to decompose subtables; Step 2.3: Identification of table element titles, column names, and data; Step 2.4: Standardize the table, regroup the sub-tables and split the merged cells; The step 2.1 includes the following specific processes: a) Use the open source plug-in to read the table information of a sheet, including the number of rows, columns, and the start and end rows. The specific method is to determine the start row or column by whether there is data in each row or column; b) Logically determine whether each row has merged cells, whether the entire row has a background color, and whether the entire row has a bold font, and write this information to the status recorder; c) Calculate the length of each row; calculate the end column information of each row. The rule of the end point is that there is no subsequent data in the cells of the current row; The step 2.2 includes the following operation process: a) Use rules to disassemble the tables in a sheet. The rules include the following points: Rule 1: The situation where the entire row of cells is merged is defined as the segmentation criteria between multiple sub-tables; Rule 2: When the font of the entire row of cells is bold, it is defined as the division standard between multiple sub-tables; Rule 3: The entire row of table data is added with a background color, and the background color of the previous row is inconsistent with the background color of the current row, which is defined as the segmentation standard between multiple sub-tables; Rule 4: Use a clustering algorithm to cluster based on the length of each row and the characteristics of the elements in the row. Based on the clustering results, when two adjacent rows are not in the same class, they are defined as the segmentation criteria between multiple sub-tables. b) Specific implementation method of clustering: Based on the DBSCAN clustering algorithm, combined with text feature input embedding information, the centroid is made to move up and down multiple times to record the distribution offset of a certain length of code. If there is an obvious offset and it is close to the offset of the current list header, the upper and lower parts are judged to be table boundaries; The step 2.3 includes the following identification method: a) Table title recognition, using rule logic, rule: merge cells in the entire row, merge downward by default, and use them as the title or note information of the current table; b) The recognition of table column names adopts rule logic. Rule 1: If the entire row is bold, it is the table column name by default. Rule 2: If the entire row has a background color, and the previous and next rows have no background color or the background color is different from the current row, it is the table column name by default. c) Table data recognition: data that does not meet the above title and column name logic rules will be assumed to be data information in the table; The step 2.4 includes the following operation process: a) Subtables are regrouped. For each subtable, rule 1: If the column name of the current table is the same as the column name of the next subtable, or the column name of one subtable completely belongs to the column name of the other subtable, the two subtables are merged; b) For each sub-table, if there are merged cells according to the table status, the merged cells are split and the split cells are assigned the same value, thereby forming one or more standard M*N tables; According to step 2.4, one or more standard M*N tables are formed; each incomplete table structure information includes: table title, table header, table data, and remarks, wherein the table title and remarks are text information of the entire row of merged cells, and the table header and table data are standard M*N data, based on the table column name and table data; they are mapped one by one with the column name in the database, and the table data mapping includes the following process: 1) Data preparation: Considering that a column of data in each table will be very long, a random sampling method is used to extract 5-10 cell data and column names from a column of data and combine them into a set of column information. The same method is used for each column of data to form N groups of corresponding prediction data; 2) Build a neural network model, use the neural network model to predict each column of data and determine its mapping relationship with the column information in the database; 3) For each table, output the final complete table structure information, including the table title, table header, table data, notes, and the mapping relationship of each column information in the database; The neural network model in step 4 includes the following encoding layers: 4.1) Encoding layer: Considering that each table contains long text, string, floating point and integer data types, and floating point, integer and string information is crucial in the financial system, it is inappropriate to use the native pre-trained model to pre-process floating point and integer numbers, so the encoding layer of the entire pre-trained model is optimized; 4.2) Numerical encoding layer: The location of the numerical value is encoded separately. According to the characteristics of the corpus, the upper limit of 5 data can be included in the numerical encoding system. Each bit of each value, including the positive and negative signs, supports 8 bits for the integer part and 4 bits for the decimal part. Each digit is losslessly one-hot encoded. Parallel to the text / string encoding, the text part is merged by the sequence mean, the highest value, and the end sequence. 4.3) Dimensionality reduction layer. Since the encoding layer has a large dimension, the text-CNN method is used here to reduce the dimension. 1D uses a low out-channel and a relatively large stride, and cooperates with a linear dimensionality reduction autoencoder to perform residual splicing to achieve dimensionality reduction. 4.4) The output layer uses a fully connected layer to access the upper feature map. Due to the dimensionality change caused by the upper layer concatenation operation, batch-regularization is used, 10% of the parameters are dropped to prevent overfitting, and then the output layer is connected to the above fully connected layer for calculation. The loss adopts multi-classification cross entropy, and the weight of the imbalanced part is adjusted; 4.5) Use the neural network model structure to map each column of data to the corresponding column in the database.