Table data extraction method based on OCR (Optical Character Recognition) multi-modal large model

Abstract

The invention provides a table data extraction method based on an OCR (Optical Character Recognition) multi-modal large model. The table data extraction method comprises the steps of data carrier acquisition and table data judgment; performing OCR (optical character recognition) on the picture format table; a large model Prompt statement is constructed; processing a large language model of the picture format table; performing multi-modal processing on the picture format table; determining a json data structure corresponding to the table data in the data carrier in the non-picture format; performing result comparison and mode selection; performing model optimization execution and field dictionary updating; carrying out model and dictionary integration testing; and carrying out system performance monitoring and self-adaptive adjustment. According to the method, the problem of format specificity in the prior art is solved, the adaptability of a tool to table data in different fields is enhanced, and the performance of a system can be continuously improved along with the increase of the processing data volume.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a table data extraction method based on an OCR multimodal large model. Background Art

[0002] Emails occupy a core position in the modern communication system, especially in formal business communication, government affairs communication and other fields. Taking the financial industry as an example, fund managers frequently transmit key data such as fund net value to investors via email, and these data are often presented in various formats such as text tables, pictures, PDF or Excel. However, current practitioners mostly rely on manual methods to check and extract table data in emails, which is inefficient and prone to errors and omissions, and it is difficult to meet the growing data processing needs.

[0003] Despite the development of automated extraction technology, significant defects are still exposed when processing email table data: Format specificity: Most existing tools are designed only for table files of specific formats, such as extraction tools specific to Excel or PDF charts. They lack cross-format collaborative processing capabilities and cannot effectively deal with the rich and diverse table formats in emails, which greatly limits their application in multi-format table data parsing scenarios. Insufficient semantic parsing: Existing tools perform poorly in terms of header semantic processing. Faced with multiple expressions of the same concept in different emails or industries, such as multiple variants of "unit net value" in the financial field, it is difficult to achieve high-precision recognition and parsing, resulting in a significant reduction in the accuracy of data extraction. Lack of domain adaptability: Existing tools generally lack a deep understanding and adaptation capability for specific fields. In the task of extracting table information in specific fields, they are unable to deeply explore the data connotation and conduct accurate data analysis, which limits their application value in professional fields. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a table data extraction method based on an OCR multimodal large model.

[0005] The present invention provides a table data extraction method based on an OCR multimodal large model, comprising:

[0006] Obtain data carriers from various data sources and determine whether the data carriers contain tabular data based on the format characteristics of the data carriers, and if so, mark the type of the data carriers;

[0007] Convert the non-image format data carrier into an image format data carrier, load the image format data carrier into the memory, call the OCR service to recognize the text, record the text position and perform error correction preprocessing, and obtain the table text information;

[0008] Construct corresponding prompt statements according to the fields to which the table data belongs, so as to add descriptions of the visual features of the images according to the image format;

[0009] Input the table text information and the prompt sentence corresponding to the table data into the pre-trained language model to extract the table data and organize it into a first json data structure, and clean the standardized data at the same time;

[0010] Input the data carrier in the image format and the prompt sentence corresponding to the table data into the OCR multimodal large model; wherein the visual recognition module of the OCR multimodal large model extracts the multi-dimensional features of the image and locates the text area, and the semantic understanding module analyzes the text semantics in combination with the visual features to extract the table data and generate a second json data structure;

[0011] Determine the json data structure corresponding to the table data in the data carrier in non-image format;

[0012] Get the number of identical JSON data structures to calculate the proportion. If the proportion exceeds the threshold, mark the corresponding processing method as the correct extraction method. If the proportion does not exceed the threshold, evaluate the accuracy of the header and data parsing of the JSON data structure of each processing method, submit the correct result and mark the wrong extraction method.

[0013] In the case where the first json data structure and the second json data structure are marked as incorrectly extracted, the corresponding table data and json data structure are corrected, the language model or the OCR multimodal model is loaded and the parameters are initialized, forward propagation calculation is performed using the corrected table data and json data structure, the loss function is calculated, the gradient is calculated by back propagation and the parameters of the language model or the OCR multimodal model are updated accordingly, multiple rounds of training and verification are performed to obtain an optimized model;

[0014] In the case where the first JSON data structure and the second JSON data structure are marked as incorrectly extracted, check the mismatch between the header representation or semantic relationship in the process and the dictionary, add or modify the dictionary content, record the updated content and reasons, and verify the validity of the test dictionary;

[0015] Set monitoring data threshold rules to analyze data feature changes or model performance degradation when the adaptive adjustment mechanism is triggered. If the data features change, modify the dictionary; if the model performance degrades, retrain the model and continue monitoring after adjustment.

[0016] Optionally, the data sources include emails, local storage and network sharing. For the email data source, JavaMail API is used to connect to the email server to filter and download email attachments; when obtaining data from the local storage, the specified folder and its subfolders are recursively traversed; for the data from the network sharing, the target file is downloaded by configuring the corresponding network protocol and access rights; when analyzing the data carrier format, the format type is determined by reading the binary data in the file header, checking the words or labels, and recording the file path, name and carrier type identification.

[0017] Optionally, the converting of the non-image format data carrier into an image format data carrier, loading the image format data carrier into a memory, calling an OCR service to recognize text, recording the text position and performing error correction preprocessing, and obtaining the table text information includes:

[0018] Use Apache POI library to open Excel file, get workbook object and traverse worksheets, analyze data distribution range to determine table boundaries and extract cell style information, and use JavaFX graphics drawing function to draw the table into PNG format image;

[0019] Use Jsoup parsing library to load Html files, locate table elements and parse their attributes and internal structure style information, use PhantomJS as Html rendering engine to render and take screenshots to generate images;

[0020] Use the PDFBox library to parse PDF files, obtain page collections, analyze page line drawing information to determine the position of table borders and cell separators and text coordinate information, and use the Java graphics drawing library to create a graphics context to draw pictures.

[0021] Optionally, when constructing the Prompt statement, common headers and their variants, data format requirements, situation handling rules, and sample data and json format conversion examples in the field are listed, and for the image format, a description of the image visual features is added, and the header description uses the initialized field header dictionary.

[0022] Optionally, the pre-trained language model is a GPT4 model. The GPT4 model extracts the table data according to the header information and data format requirements in the Prompt statement and organizes it into a first json data structure, and cleans and standardizes the data, including removing redundant punctuation and converting the date format.

[0023] Optionally, the visual recognition module of the OCR multimodal large model extracts multi-dimensional features of the image including table shape, text font style and image color distribution, and accurately locates the text area; the semantic understanding module analyzes the text semantics in combination with the visual features, determines the table header category according to the text position, understands the data association based on the logical relationship of the table, extracts data to generate a second json data structure, and compares and verifies or complements the first json data structure.

[0024] Optionally, the step of determining a JSON data structure corresponding to the table data in the data carrier in a non-picture format includes:

[0025] For data carriers in Excel format, grab table data, determine the table direction, transpose if it is vertical and handle references and formats, build or connect domain dictionaries / models, map header information to a unified format, traverse data rows to build a third json data structure and clean and convert data;

[0026] For data carriers in Html format, load table data, parse structure and style information, adjust non-horizontal table direction, map table header semantics with the help of dictionary / model, extract data after processing complex table header situations to construct the fourth json data structure, process text encoding and clean standard data;

[0027] For data carriers in PDF format, parse table files, analyze page layout to determine direction and correct it, use dictionary / model mapping to unify header expressions, process complex headers, read data to build the fifth json data structure and clean and convert data.

[0028] Optionally, use the EasyExcel tool to capture Excel table data, determine the table direction and transpose the vertical table, build a dictionary / model mapping header information in the target field to a unified format, traverse the data rows to build a third json data structure and clean and convert the data, including date format conversion and numerical data precision adjustment.

[0029] Optionally, use Jsoup to load Html format table data, parse structure and style information, adjust non-horizontal table direction, use dictionary / model in target domain to map header semantics, handle complex header situations, extract data to construct fourth json data structure, process text encoding and clean standard data, remove Html tag residues and extra blank characters.

[0030] Optionally, use PDFBox to parse PDF format table files, analyze the page layout to determine the table direction, process and correct the direction page by page, use the dictionary / model mapping in the target field to unify the header expression, process complex headers, read data to build the fifth json data structure, and clean and convert the data.

[0031] The present invention provides a table data extraction method based on an OCR multimodal large model, a complete link from various formats (Html, pictures, Excel and PDF) to a unified json data structure extraction, and a specific tool and model combination method adopted for different formats, such as two parallel processing model architectures for pictures and tables, which can effectively realize cross-format collaborative work and overcome the problem of format specificity in the prior art.

[0032] By deeply integrating dictionaries that carry rich semantic information and domain knowledge with OCR technology and large language models, the constructed Prompt statement guides the model to perform high-precision semantic analysis, which can accurately identify various expressions of header semantics and improve the accuracy of data extraction.

[0033] The application of domain dictionaries and domain big models in the processing of tables of different formats, as well as the dynamic expansion and correction mechanism of dictionaries based on manual review results, effectively enhance the tool's adaptability to tabular data in different fields.

[0034] The strategy of combining automatic comparison with manual review, as well as the feedback mechanism based on the review results, including model training data updates for image table processing and dictionary corrections for table processing in other formats, achieves self-optimization of the extraction process and enables the system to continuously improve performance as the amount of processed data increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a table data extraction method based on an OCR multimodal large model is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, this embodiment provides a table data extraction method based on an OCR multimodal large model, including:

[0039] Step 101, obtain data carriers from various data sources and determine whether the data carrier contains table data according to the format characteristics of the data carrier, and if so, mark the type of the data carrier.

[0040] In this step, the data sources include emails, local storage, and network shares. For the email data source, the JavaMail API is used to connect to the email server, and the email attachments are filtered and downloaded according to the set rules (such as specific senders, subject keywords, etc.). These attachments may contain the required table data carriers. When obtaining data from local storage, the specified folder and its subfolders are recursively traversed to identify files of various formats. For data from network shares, the target file is downloaded by configuring the corresponding network protocol (such as SMB protocol) and access rights; when analyzing the data carrier format, the format type is determined by reading the binary data in the file header, checking the words or labels, and recording the file path, name, and carrier type identifier.

[0041] When analyzing the data carrier format, for the image format, read the binary data in the file header and compare it with the characteristic identifiers of common image formats (such as FF D8 FF for JPEG, 89 50 4E 47 for PNG, etc.) to determine whether it is an image and the specific image format type. For PDF files, check whether there is "% PDF" in the file header. For Excel files, in addition to checking the extension, the specific byte sequence in the file header is parsed to confirm whether it is a valid Excel format. For Html files, the initial judgment is based on whether there is a "" tag. Once it is determined that it is a carrier that may contain table data, its file path, name and corresponding carrier type identifier are recorded in the database so that the subsequent processing flow can quickly locate and call it.

[0042] Step 102, converting the non-image format data carrier into an image format data carrier, loading the image format data carrier into the memory, calling the OCR service to recognize the text, recording the text position and performing error correction preprocessing, and obtaining the table text information.

[0043] In this step, error correction preprocessing includes correcting common typos, identifying incorrect punctuation marks, etc.

[0044] After opening the Excel file using the Apache POI library, first obtain the workbook object and then traverse each worksheet. Determine the boundaries of the table by analyzing the distribution range of the data in the worksheet, and extract the style information of the cell, including font, font size, color, border style, etc. Use the graphics drawing function of JavaFX to create a graphics scene that matches the size of the table, draw each cell to the corresponding position in the graphics scene in turn according to the style and content of the cell, and finally save the graphics scene as a PNG format image file. During the drawing process, ensure that the text alignment, line break processing, etc. are consistent with the original Excel table to ensure the complete and accurate presentation of the table information.

[0045] Use the Jsoup parsing library to load the Html file and accurately locate the table element through the selector syntax ( ). Parse table element attributes (such as width, height, cellpadding, etc.) and internal rows ( ) Sum column ( ) element structure and style information, including text content, links, style classes, etc. Use PhantomJS as the Html rendering engine to create a headless browser environment, render the parsed table data on the page in the original style, then take a screenshot of the page and save it as an image file. During the rendering process, set the appropriate screenshot area and resolution according to the actual size of the table and the page layout to ensure that the generated image clearly displays all the details of the table.

[0046] With the help of PDFBox library to parse PDF files, we first get the page set in the document, and then analyze each page in detail. By detecting the line drawing information in the page, we can determine the position of the table border and cell separator, and extract the text content and its coordinate information. Based on this information, we use Java's graphics drawing library (such as Graphics2D) to create a graphics context, draw the table border, text and other elements into the graphics context in turn, and generate an image file. During the processing, for complex situations such as rotation, scaling, and cell merging in PDF, we use algorithms such as coordinate transformation and text merging to accurately process and restore the image consistent with the original PDF table layout.

[0047] Step 103, construct a corresponding Prompt statement according to the field to which the table data belongs, so as to add a description of the visual features of the picture according to the picture format.

[0048] In this step, when constructing the Prompt statement, common headers and their variants, data format requirements, situation handling rules, and sample data and json format conversion examples in the field are listed. For the image format, a description of the image visual features is added, and the header description uses the initialized field header dictionary.

[0049] Take fund product notification emails as an example, and construct Prompt statements for the fund field: In fund product net value notifications, common table headers include "date", "unit net value", "cumulative net value", "daily growth rate", etc. Date variants may include "net value date" and "business date"; unit net value variants may include "fund share net value"; cumulative net value variants may include "fund cumulative net value"; daily growth rate variants may include "daily growth rate", etc. In terms of data format requirements, the date format is "YYYY-MM-DD", the unit net value and cumulative net value are retained to four decimal places, and the daily growth rate is expressed as a percentage and retains two decimal places. Special case handling rules For example, when there is no transaction on a certain day, the unit net value and cumulative net value are the same as the previous trading day, and the marked daily growth rate is "0.00%". The sample data is "2024-11-15, 1.0567, 2.3456, 1.23%", and the json format conversion example is {"Date":"2024-11-15","Unit Net Value":"1.0567","Cumulative Net Value":"2.3456","Daily Growth Rate":"1.23%"}. For the image format, add a description of the image's visual features. For example, if there is a data table in the image, the table lines are black, the text color is dark gray, the background is white, and the header text font is large and bold. The description of the header will use the initialized header dictionary of the field.

[0050] Step 104, input the table text information and the prompt sentence corresponding to the table data into the pre-trained language model to extract the table data and organize it into a first json data structure, and clean the standardized data at the same time.

[0051] In this step, the pre-trained language model is the GPT4 model. The GPT4 model extracts the table data after the image OCR recognition according to the header information and data format requirements in the Prompt statement and organizes it into the first json data structure. At the same time, the data is cleaned and standardized, including removing redundant punctuation and converting the date format. For example, the identified fund net value data is sorted according to the specified json format, and the "unit net value" data is accurately placed in the corresponding json field. At the same time, the data is cleaned and standardized, the redundant punctuation in the text is removed, and the irregular date format such as "2023.01.15" is converted to "2023-01-15".

[0052] Step 105, input the data carrier in the image format and the Prompt statement corresponding to the table data into the OCR multimodal large model; wherein the visual recognition module of the OCR multimodal large model extracts the multi-dimensional features of the image and locates the text area, and the semantic understanding module analyzes the text semantics in combination with the visual features to extract the table data to generate a second json data structure.

[0053] In this step, the visual recognition module of the OCR multimodal large model extracts the multi-dimensional features of the image, including the table shape, text font style and image color distribution, and accurately locates the text area, such as accurately finding the area where each net value data is located; the semantic understanding module analyzes the text semantics in combination with the visual features, determines the table header category to which the text belongs according to its position in the table, understands the data association based on the table logical relationship, extracts the data to generate a second json data structure, and compares and verifies or complements it with the first json data structure. For example, the first row of the table is identified as the header through visual features, and then the second row and subsequent data are extracted and generated in accordance with the header definition in combination with semantic understanding. JSON data is compared and verified or complemented with the processing results in step 104 to improve the accuracy of data extraction.

[0054] Step 106, determining the JSON data structure corresponding to the table data in the data carrier in the non-picture format.

[0055] In this step, for the data carrier in Excel format, the table data is captured, the table direction is determined, if it is vertical, it is transposed and the references and formats are processed, the domain dictionary / model is built or connected, the header information is mapped to a unified format, the data rows are traversed to build a third json data structure and the converted data is cleaned.

[0056] For Excel table carriers in email attachments, use the EasyExcel tool to capture Excel table data. Determine the direction of the table. If it is found that the net value data in the Excel table is arranged vertically (arranged by column) (such as one column is the date, and the other column is the unit net value, etc.), use the transposition function of the EasyExcel tool to convert it into a horizontal table, and handle the relevant cell references and format adjustments after the transposition to ensure the accuracy and completeness of the data. Build a fund field dictionary / model to map header information such as "Fund Share Net Value" to a unified format "Unit Net Value". Then traverse the data rows and build a json data structure to convert each row of data in Excel into an element in a json object. At the same time, clean and convert the data, such as converting the data in the date column to a standard date format, and adjust the precision of the numerical data to make it meet the format requirements for the net value data in the json data.

[0057] For data carriers in Html format, load table data, parse structure and style information, adjust non-horizontal table direction, map header semantics with the help of dictionary / model, extract data after processing complex header situations to construct the fourth json data structure, process text encoding and clean and standardize data.

[0058] For the HTML table carrier in the body of the email, Jsoup is used to load the HTML table data and parse its structure and style information, such as obtaining the number of rows and columns in the table, the merging of cells, style class names, etc. If the table is not horizontal (such as the net value data is arranged in rows), adjust its direction to horizontal for subsequent processing. Use the fund field dictionary / model to map the header semantics, for example, map "net value change amount" to a unified semantic identifier, so that there is a unified field name in the json data structure. Handle complex header situations, such as those with nested headers or merged cell headers, extract data to build a json data structure, and handle text encoding issues, such as converting possible garbled characters into correct encoded characters, while cleaning and standardizing data, removing HTML tag residues, redundant blank characters, etc., to ensure the purity and accuracy of the net value data.

[0059] For data carriers in PDF format, parse table files, analyze page layout to determine direction and correct it, use dictionary / model mapping to unify header expressions, process complex headers, read data to build the fifth json data structure and clean and convert data.

[0060] For the PDF table carrier in the attachment, use PDFBox to parse the table file, analyze the page layout and determine the direction of the table. In the PDF file, there may be multiple pages of net value data tables, which need to be processed page by page and corrected in direction. Use the fund field dictionary / model to unify the header expression, for example, map the header in the "Fund Net Value Change Status Table" with the standard fund net value header expression. Process complex headers, such as those with multi-level headers or headers with comments, read data to build a json data structure, and clean and convert the data during the reading process, such as removing the currency symbol in the amount data, converting the percentage data to decimal form and retaining the corresponding decimal places according to the json data structure requirements to ensure the standardization of the data in the json data structure.

[0061] Step 107, obtain the number of identical JSON data structures to calculate the proportion. When the proportion exceeds the proportion threshold, mark the corresponding processing method as the correct extraction method. When the proportion does not exceed the proportion threshold, evaluate the header and data parsing accuracy of the JSON data structure of each processing method, submit the correct result and mark the wrong extraction method.

[0062] Collect the json data structure results obtained in step 104 (language large model processing of picture format tables), step 105 (multimodal processing flow of picture format tables), and step 106 (original format processing of Excel format tables, original format processing of Html format tables, and original format processing of PDF format tables).

[0063] Compare these json data structure results, count the number of identical json data structures and calculate their proportion. Assume that the proportion threshold is set to 60%. If the proportion of identical json data structures is greater than 60%, enter the automatic selection mode; if it is less than or equal to 60%, enter the manual review mode of S12. For example, if it is found through statistics that the results of step 104 and step 105 are mostly the same and the proportion reaches 70%, enter the automatic selection mode; if the results of each processing method are quite different and the proportion of the same part is only 40%, enter the manual review mode.

[0064] Output the json data structure with a frequency greater than or equal to the threshold (60%), mark the corresponding processing methods (such as step 104 and step 105) as the correct extraction method, and mark other processing methods (such as the part with a large difference from the high-frequency result in step 106) as the wrong extraction method. For example, if the json format results of the fund net value data extracted in step 104 and step 105 are consistent and the proportion exceeds the threshold, but the results are slightly different due to some special circumstances in the Excel format conversion process, then mark the extraction methods of step 104 and step 105 as correct, and the extraction method of step 106 as incorrect.

[0065] Manually comprehensively evaluate the accuracy of the header and data parsing of the json data structure results from step 104 to step 106. For example, manually check whether the fund net value date is incorrectly recognized when the language model processes the image format table in step 104, and whether the complex formula calculation results are correctly parsed when the Excel format is processed in step 106. Manually submit the correct results and mark the wrong extraction method. For example, if the header mapping error is found when the Html format is processed in step 106, it will be marked as an incorrect extraction method for subsequent optimization.

[0066] Step 108, when the first json data structure and the second json data structure are marked as incorrectly extracted, correct the corresponding table data and json data structure, load the language model or the OCR multimodal model and initialize the parameters, use the corrected table data and json data structure to perform forward propagation calculations, calculate the loss function, back propagate and calculate the gradients and update the parameters of the language model or the OCR multimodal model accordingly, perform multiple rounds of training and verification, and obtain an optimized model.

[0067] For the case where step 104 or step 105 is marked as an incorrect extraction method:

[0068] The manually reviewed and corrected table data and the correct json data are combined into a training data pair. For example, if the fund net asset value table image is incorrectly identified in step 104, the manually corrected net asset value data and the correct json format net asset value data are combined into a training data pair.

[0069] Clean the data pairs, remove noise data, such as interfering lines and blurred spots in the image that cause incorrectly identified data, standardize the data format, unify the data into the format required for model training, check the integrity and mark missing data, such as if some fund net value data in the image is missing, mark it in the data pair.

[0070] Construct a data pair set to ensure the diversity and representativeness of the data pairs, including training data pairs of different fund products, different table styles, and different data sizes, and divide the data into small batches for training. For example, divide 800 data pairs into batches of 100 data pairs.

[0071] For the training set, load the language model or OCR multimodal model and initialize the parameters. Perform forward propagation calculations on small batches of data, and output json structured data after the model processes the data. Calculate the loss function and compare the output data with the correct json data, such as calculating the error between the fund net value data output by the model and the correct net value data. Backward propagation calculates the gradient and updates the model parameters accordingly. Perform multiple rounds of training (such as 8 rounds) and verify. Use verification methods such as the leave-one-out method to avoid overfitting, and obtain an optimized model, so that the accuracy of the model is improved when processing image format table data in the fund field.

[0072] Step 109, when the first JSON data structure and the second JSON data structure are marked as incorrectly extracted, check whether the header expression or semantic relationship in the process does not match the dictionary, add or modify the dictionary content, record the updated content and reasons, and verify the validity of the test dictionary.

[0073] In this step, for the case where steps other than step 104 and step 105 are marked as incorrect extraction methods:

[0074] Manual inspection is performed to check if the table header expression or semantic relationship does not match the dictionary. For example, when processing an Excel format table in step 106, it is found that the table header "Fund Net Asset Value" is not accurately mapped in the dictionary, or when processing an Html format table, the semantic relationship of "Fund Net Asset Value Growth Rate" is not accurately defined in the dictionary.

[0075] Dictionary modification: Add or modify dictionary content, such as adding "Fund Net Asset Value" to the dictionary and associating the correct semantics and data types, and modifying the semantic relationship definition of "Fund Net Asset Value Growth Rate". Record the updated content and reasons, such as "Due to the appearance of new net asset value-related terms in the Excel table, update the dictionary to adapt". Verify the validity of the test dictionary and use some test data to verify whether the updated dictionary can correctly map the table header and data when processing related tables.

[0076] Step 1010, set monitoring data threshold rules, analyze data feature changes or model performance degradation when the adaptive adjustment mechanism is triggered, if the data features change, modify the dictionary, if the model performance degrades, retrain the model, and continue monitoring after adjustment.

[0077] If the optimization of step 108 or step 109 is executed, the optimization model and updated dictionary will be integrated into the system. Select test table data carriers of various formats and fund fields for processing, such as selecting new fund product net value notification emails (including tables in different formats), new Excel net value calculation files, new PDF net value fluctuation analysis reports, etc. Compare the processing results with the correct results of manual annotation, count the number and types of errors, such as the number of fund net value data extraction errors, header parsing errors, etc., and calculate evaluation indicators (such as accuracy, recall, etc.). If the performance does not meet expectations, return to step 108 (if it is a problem related to image format table processing) or step 109 (if it is a problem related to other format table header mapping problems) according to the carrier type and adjust until the requirements are met, such as requiring the accuracy rate to reach more than 90%.

[0078] After the system is deployed, performance indicators are continuously monitored, such as the time to process a fund product net asset value notification email, the success rate and error rate of extracting data from tables in different formats, etc. Threshold rules are set based on the monitoring data, such as setting a processing time of more than 8 seconds as an abnormality and an error rate of more than 3% as an abnormality. When the adaptive adjustment mechanism is triggered, analyze the changes in data features or the decline in model performance. If it is found that the format of the fund product net asset value notification email has changed significantly (such as the addition of a special table layout), resulting in changes in data features, enter dictionary correction; if the model's performance declines after processing a large amount of fund net asset value data (such as a decrease in accuracy), enter model training, and continue monitoring after adjustment to ensure that the system always maintains a high performance level and accurately extracts table data from fund product net asset value notification emails.

[0079] In summary, the table data extraction method based on the OCR multimodal large model provided in this embodiment provides a complete link from various formats (Html, pictures, Excel and PDF) to a unified json data structure extraction, as well as specific tool and model combinations adopted for different formats, such as two parallel processing model architectures for pictures and tables, which can effectively realize cross-format collaborative work and overcome the problem of format specificity in the prior art.

[0080] Semantic parsing and dictionary-model fusion mechanism protection: By deeply integrating dictionaries that carry rich semantic information and domain knowledge with OCR technology and large language models, the constructed Prompt statement guides the model to perform high-precision semantic parsing, which can accurately identify various expressions of header semantics and improve the accuracy of data extraction.

[0081] Technical protection for domain adaptability enhancement: The application of domain dictionaries and domain big models in the processing of tables in different formats, as well as the dynamic expansion and correction mechanism of dictionaries based on manual review results, effectively enhance the tool's adaptability to table data in different fields.

[0082] Self-optimization feedback mechanism protection: The strategy of combining automatic comparison with manual review, as well as the feedback mechanism based on the review results, including model training data updates for image table processing and dictionary corrections for other format table processing, realizes the self-optimization of the extraction process. It enables the system to continuously improve performance as the amount of data processed increases.

[0083] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A table data extraction method based on OCR multimodal large model, characterized in that: include: Obtain data carriers from various data sources and determine whether the data carriers contain tabular data based on the format characteristics of the data carriers, and if so, mark the type of the data carriers; Convert the non-image format data carrier into an image format data carrier, load the image format data carrier into the memory, call the OCR service to recognize the text, record the text position and perform error correction preprocessing, and obtain the table text information; Construct corresponding prompt statements according to the fields to which the table data belongs, so as to add descriptions of the visual features of the images according to the image format; Input the table text information and the prompt sentence corresponding to the table data into the pre-trained language model to extract the table data and organize it into a first json data structure, and clean the standardized data at the same time; Input the data carrier in the image format and the prompt sentence corresponding to the table data into the OCR multimodal large model; wherein the visual recognition module of the OCR multimodal large model extracts the multi-dimensional features of the image and locates the text area, and the semantic understanding module analyzes the text semantics in combination with the visual features to extract the table data and generate a second json data structure; Determine the json data structure corresponding to the table data in the data carrier in non-image format; Get the number of identical JSON data structures to calculate the proportion. If the proportion exceeds the threshold, mark the corresponding processing method as the correct extraction method. If the proportion does not exceed the threshold, evaluate the accuracy of the header and data parsing of the JSON data structure of each processing method, submit the correct result and mark the wrong extraction method. In the case where the first json data structure and the second json data structure are marked as incorrectly extracted, the corresponding table data and json data structure are corrected, the language model or the OCR multimodal model is loaded and the parameters are initialized, forward propagation calculation is performed using the corrected table data and json data structure, the loss function is calculated, the gradient is calculated by back propagation and the parameters of the language model or the OCR multimodal model are updated accordingly, multiple rounds of training and verification are performed to obtain an optimized model; In the case where the first JSON data structure and the second JSON data structure are marked as incorrectly extracted, check the mismatch between the header representation or semantic relationship in the process and the dictionary, add or modify the dictionary content, record the updated content and reasons, and verify the validity of the test dictionary; Set monitoring data threshold rules to analyze data feature changes or model performance degradation when the adaptive adjustment mechanism is triggered. If the data features change, modify the dictionary; if the model performance degrades, retrain the model and continue monitoring after adjustment.

2. The table data extraction method based on the OCR multimodal large model according to claim 1 is characterized in that: The data sources include emails, local storage and network sharing. For the email data source, JavaMail API is used to connect to the email server to filter and download email attachments; when obtaining data from the local storage, the specified folder and its subfolders are recursively traversed; for the data from the network sharing, the target file is downloaded by configuring the corresponding network protocol and access rights; when analyzing the data carrier format, the format type is determined by reading the binary data in the file header, checking the words or labels, and recording the file path, name and carrier type identification.

3. The table data extraction method based on the OCR multimodal large model according to claim 1 is characterized in that: The method converts the non-image format data carrier into an image format data carrier, loads the image format data carrier into the memory, calls the OCR service to recognize the text, records the text position and performs error correction preprocessing, and obtains the table text information, including: Use Apache POI library to open Excel file, get workbook object and traverse worksheets, analyze data distribution range to determine table boundaries and extract cell style information, and use JavaFX graphics drawing function to draw the table into PNG format image; Use Jsoup parsing library to load Html files, locate table elements and parse their attributes and internal structure style information, use PhantomJS as Html rendering engine to render and take screenshots to generate images; Use the PDFBox library to parse PDF files, obtain page collections, analyze page line drawing information to determine the position of table borders and cell separators and text coordinate information, and use the Java graphics drawing library to create a graphics context to draw pictures.

4. The table data extraction method based on the OCR multimodal large model according to claim 1 is characterized in that: When constructing the Prompt statement, list the common headers and their variants, data format requirements, situation handling rules, and sample data and json format conversion examples in the field. For the image format, add a description of the image visual features, and the header description uses the initialized field header dictionary.

5. The table data extraction method based on OCR multimodal large model according to claim 1 is characterized in that: The pre-trained language model is the GPT4 model. The GPT4 model extracts the table data based on the header information and data format requirements in the Prompt statement and organizes it into the first json data structure. At the same time, the data is cleaned and standardized, including removing redundant punctuation and converting the date format.

6. The table data extraction method based on OCR multimodal large model according to claim 1 is characterized in that: The visual recognition module of the OCR multimodal large model extracts multi-dimensional features of the image, including table shape, text font style and image color distribution, and accurately locates the text area; the semantic understanding module analyzes the text semantics based on visual features, determines the table header category based on the text position, understands the data association based on the logical relationship of the table, extracts data to generate a second json data structure, and compares and verifies it with the first json data structure or complements each other.

7. The table data extraction method based on OCR multimodal large model according to claim 1 is characterized in that: The step of determining the json data structure corresponding to the table data in the data carrier in the non-picture format includes: For data carriers in Excel format, grab table data, determine the table direction, transpose if it is vertical and handle references and formats, build or connect domain dictionaries / models, map header information to a unified format, traverse data rows to build a third json data structure and clean and convert data; For data carriers in Html format, load table data, parse structure and style information, adjust non-horizontal table direction, map table header semantics with the help of dictionary / model, extract data after processing complex table header situations to construct the fourth json data structure, process text encoding and clean standard data; For data carriers in PDF format, parse table files, analyze page layout to determine direction and correct it, use dictionary / model mapping to unify header expressions, process complex headers, read data to build the fifth json data structure and clean and convert data.

8. The table data extraction method based on the OCR multimodal large model according to claim 7 is characterized in that: Use the EasyExcel tool to capture Excel table data, determine the table direction and transpose the vertical table, build a dictionary / model mapping header information in the target field to unify the format, traverse the data rows to build the third json data structure and clean and convert the data, including date format conversion and numerical data precision adjustment.

9. The table data extraction method based on OCR multimodal large model according to claim 7 is characterized in that: Use Jsoup to load Html format table data, parse structure and style information, adjust non-horizontal table direction, use dictionary / model in target domain to map header semantics, handle complex header situations, extract data to build fourth json data structure, process text encoding and clean standard data, remove Html tag residues and redundant blank characters.

10. The table data extraction method based on OCR multimodal large model according to claim 7, characterized in that: Use PDFBox to parse PDF table files, analyze the page layout to determine the direction of the table, process and correct the direction page by page, use the dictionary / model mapping in the target field to unify the header expression, process complex headers, read data to build the fifth json data structure, and clean and convert the data.

Images