Multi-modal data preprocessing and fusion technology and system based on artificial intelligence

Abstract

The invention provides a multi-modal data preprocessing and fusion technology and system based on artificial intelligence, and relates to the technical field of data processing, and the technology comprises the steps: converting a PDF document into a table image, and carrying out the preprocessing; a large visual model is adopted for table structure analysis, and zero sample understanding is achieved through a prompt text; finely tuning the model based on the synthetic data set; generating standardized output by using the large language model; and the training lightweight model is deployed on edge equipment to realize offline processing. According to the method, the table identification accuracy is improved, the resource consumption is reduced, and efficient identification and processing of the complex table are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-modal data preprocessing and fusion technology and system based on artificial intelligence. BACKGROUND

[0002] The ship industry is a comprehensive system with multi-industry and multi-field cooperation. The ability control object is complex, the control range is wide, the control requirement is high, the ability data is scattered, the data attribute dimension is multiple, the data structure is complex, the effective transmission path of the data along the management chain is not smooth, the data "island" phenomenon and fragmentation phenomenon are obvious, and the current data systematization collection, arrangement and fusion have not been realized. It is urgent to carry out multi-modal data preprocessing and fusion technology research based on the characteristics of the ship industry.

[0003] In view of the characteristics of wide data sources, scattered data, various formats and multiple attributes in the ship industry, a multi-modal data analysis engine is constructed based on natural language processing and computer vision technology, realizing the identification and capture of massive heterogeneous and different modal ship product, ability and other original data (including common documents (PDF, word, picture, audio, video), tables (Excel, CSV), pure text format (TXT) and other formats, etc.), forming product, ability, equipment attribute and other characteristic information. Using the leading feature extraction method, the extracted information is converted into a feature vector that can be used by machine learning algorithms, further improving the quality and usability of the data. According to the characteristics of the ship industry, a suitable open source large model is selected, and through pre-training and fine-tuning, a special model for ship industry product and ability monitoring and evaluation data governance is formed, and detailed tables of equipment information, project library information and product information are output, providing clear and accurate data support for modeling, analysis and decision-making.

[0004] Tables are common forms of information organization, widely used in business reports, financial data, scientific literature and government documents. With the advancement of digital transformation, a large number of paper documents are converted into PDF format, which contains a large amount of table data. Accurate and efficient extraction and understanding of table structure and content from PDF documents is of great significance for data analysis, automation and knowledge management.

[0005] Traditional table recognition methods mainly rely on rules and template matching, which are not effective in dealing with complex structure and variable format tables. In recent years, with the development of computer vision and natural language processing technology, deep learning-based table recognition methods have made significant progress. In particular, the emergence of visual large models and language large models provides a new technical path for multi-modal table understanding.

[0006] Existing methods have limited understanding of complex table structures, especially for tables with cell merging, nested headers, or non-standard formats, and have low recognition accuracy, often requiring specialized training for specific domain tables to achieve good results.

[0007] Most table recognition systems lack effective semantic understanding capabilities and can only extract table structures and text content, but cannot reason and verify formula relationships in the table, leading to logical errors or data inconsistencies in the extraction results.

[0008] Existing large model-based table recognition methods require high computational resources and are difficult to efficiently deploy and run on edge devices, limiting their application scenarios in resource-constrained environments and failing to meet offline processing and real-time analysis requirements. SUMMARY

[0009] The embodiments of the present application provide a multi-modal data preprocessing and fusion technology and system based on artificial intelligence, which can solve the problems in the prior art.

[0010] In a first aspect, the embodiments of the present application provide a multi-modal data preprocessing and fusion technology based on artificial intelligence, comprising: Converting an input PDF document into a table image, preprocessing the table image, including table region detection, direction correction, image enhancement, noise reduction processing, and border line removal, to obtain a preprocessed table image; Performing structure analysis on the preprocessed table image using a visual large model, inputting a prompt text containing table structure extraction instructions to the visual large model, the prompt text being used to guide the visual large model to output structure description information of the table, the structure description information including row numbers, column numbers, merging information, and contents of each cell in the table, and realizing zero-shot table structure understanding based on the prompt text; Constructing a synthetic table dataset, fine-tuning the visual large model based on the synthetic table dataset, and realizing efficient updating of model parameters using low-rank adaptation technology; inputting the structure description information output by the visual large model into a language large model, and performing semantic extraction, formula deduction and verification by the language large model to generate standardized table structure output; Using the trained visual large model as a first teacher model to train a lightweight visual model, using the trained language large model as a second teacher model to train a lightweight language model, deploying the lightweight visual model and the lightweight language model on an edge device, and realizing offline processing of tables through the lightweight visual model and the lightweight language model.

[0011] The visual large model is used for structural analysis on the preprocessed table image, prompt text containing table structure extraction instructions is input into the visual large model, and the prompt text includes: The visual large model is used for structural analysis on the preprocessed table image, and the visual large model extracts structural features of the table from the table image through a multi-head attention mechanism; The prompt text containing table structure extraction instructions is input into the visual large model, the prompt text contains table row and column recognition instructions, cell merging detection instructions, and content extraction instructions, and the prompt text is used to guide the visual large model to understand the table structure; Based on the prompt text, the visual large model analyzes the structural features, and outputs a table structure description result containing cell position information, merging information and content information from the visual large model.

[0012] A synthetic table dataset is constructed, the visual large model is fine-tuned based on the synthetic table dataset, and efficient updating of model parameters is realized by using a low-rank adaptation technique, including: The synthetic table dataset is constructed, and the synthetic table dataset contains table samples with multiple levels of table headers, merged cells and nested structures, and random numerical content of the table samples is generated using structured rules; The visual large model is fine-tuned based on the synthetic table dataset, the table samples are input into the visual large model, and the structure recognition loss of the visual large model is calculated based on the true structure information of the table samples; The parameters of the visual large model are updated using a low-rank adaptation technique, the number of trainable parameters is reduced by a low-rank matrix decomposition method, and efficient updating of the parameters of the visual large model is realized.

[0013] The structure description information output by the visual large model is input into the language large model, semantic extraction, formula derivation and verification are performed by the language large model, and a standardized table structure output is generated, including: The structure description information output by the visual large model is input into the language large model, the structure description information contains cell position information, cell merging information and cell content information of the table, and the language large model performs semantic understanding on the structure description information; Based on the language large model, semantic extraction and formula derivation are performed, the language large model identifies numerical relationships and calculation formulas in the table, verifies the correctness of the numerical relationships and the calculation formulas, and ensures the logical consistency of the table data, and the language large model generates a standardized table structure output.

[0014] The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises: The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises: The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises:

[0015] The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises: The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises: The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises: The trained visual large model is taken as a first teacher model to train a lightweight visual model, and the trained language large model is taken as a second teacher model to train a lightweight language model, and the training of the lightweight visual model and the lightweight language model comprises:

[0016] The second aspect of the embodiment of the application provides a multi-modal data preprocessing and fusion system based on artificial intelligence, which comprises: The first unit is used for converting an input PDF document into a table image, and pre-processing the table image, including table region detection, direction correction, image enhancement, noise reduction processing and frame line removal, to obtain a pre-processed table image. The second unit is used for performing structure analysis on the pre-processed table image by using a visual large model, inputting a prompt text containing table structure extraction instructions into the visual large model, the prompt text being used to guide the visual large model to output structure description information of the table, the structure description information including row numbers, column numbers, merging information and contents of each cell in the table, and realizing zero-shot table structure understanding based on the prompt text. The third unit is configured to construct a synthetic table dataset, fine-tune the visual large model based on the synthetic table dataset, and realize efficient updating of model parameters by using a low-rank adaptation technology; and input structural description information output by the visual large model into a language large model, perform semantic extraction, formula deduction and verification by the language large model, and generate a standardized table structure output. The fourth unit is configured to train a lightweight visual model by taking the trained visual large model as a first teacher model, train a lightweight language model by taking the trained language large model as a second teacher model, and deploy the lightweight visual model and the lightweight language model on an edge device to realize offline processing of a table by the lightweight visual model and the lightweight language model.

[0017] In a third aspect, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0018] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0019] The present application has the following beneficial effects: The present application realizes high-precision table structure and content recognition through the collaborative work of the visual large model and the language large model, especially for tables with complex layouts and low-quality images, which can maintain good recognition performance, effectively solving the problem of complex scenes that traditional table recognition methods cannot handle.

[0020] The present application uses a low-rank adaptation technology to efficiently fine-tune the visual large model, so that the model can adapt to different table styles and structural characteristics in different fields, while significantly reducing the training resource consumption, improving the recognition accuracy of the model in a specific field, and realizing efficient model adaptation.

[0021] The present application trains lightweight visual and language models by a knowledge distillation method and deploys them on edge devices, so that the system can perform offline table processing without relying on network connection, which not only ensures data security, but also greatly reduces hardware resource demand, improves the practicability and universality of the system. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a multi-modal data preprocessing and fusion technology based on artificial intelligence according to an embodiment of the present application is shown in the figure. Figure 2 A workflow diagram for table image structure analysis of embodiments of the present application. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0024] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0025] Figure 1 A process schematic diagram of the multi-modal data preprocessing and fusion technology based on artificial intelligence of embodiments of the present application is shown as follows. Figure 1 The method comprises: Converting an input PDF document into a table image, preprocessing the table image, including table region detection, direction correction, image enhancement, noise reduction processing, and border line removal, to obtain a preprocessed table image; Performing structure analysis on the preprocessed table image by using a visual large model, inputting a prompt text containing table structure extraction instructions into the visual large model, the prompt text being used to guide the visual large model to output structure description information of the table, the structure description information including row numbers, column numbers, merging information, and contents of each cell in the table, and realizing zero-shot table structure understanding based on the prompt text; Constructing a synthetic table dataset, fine-tuning the visual large model based on the synthetic table dataset, and realizing efficient updating of model parameters by using low-rank adaptation technology; inputting the structure description information output by the visual large model into a language large model, performing semantic extraction, formula deduction, and verification by the language large model, and generating standardized table structure output; Training a lightweight visual model by taking the trained visual large model as a first teacher model, training a lightweight language model by taking the trained language large model as a second teacher model, deploying the lightweight visual model and the lightweight language model on an edge device, and realizing offline processing of the table by the lightweight visual model and the lightweight language model.

[0026] In an optional implementation, a large visual model is used to perform structural analysis on the preprocessed table image, and inputting a prompt text containing table structure extraction instructions to the large visual model includes: The large visual model is used to perform structural analysis on the preprocessed table image, and the large visual model extracts structural features of the table from the table image through a multi-head attention mechanism; The prompt text containing table structure extraction instructions is input to the large visual model, and the prompt text contains table row and column recognition instructions, cell merging detection instructions, and content extraction instructions, and the prompt text is used to guide the large visual model to understand the table structure; Based on the prompt text, the large visual model analyzes the structural features, and outputs a table structure description result containing cell position information, merging information and content information from the large visual model.

[0027] As shown in Figure 2 The method includes: The large visual model is used to perform structural analysis on the preprocessed table image, and the large visual model is a deep learning model combining image understanding and natural language processing capabilities, which can process visual information and text information simultaneously. The model extracts structural features of the table from the table image through a multi-head attention mechanism. The multi-head attention mechanism allows the model to learn the relevance of different positions in different representation subspaces. By dividing the table image into multiple regions and calculating the attention weights between these regions, the model can capture the spatial relationships between cells in the table and the arrangement patterns of rows and columns.

[0028] In practical applications, the multi-head attention calculation process of the large visual model involves mapping input image features to query, key and value representations, then calculating the similarity between query and key to obtain attention weights, and finally applying these weights to the values to obtain weighted features. By setting 8 attention heads, each head focuses on different feature spaces, the model can comprehensively capture horizontal and vertical structural clues of the table, identify cell boundaries, row and column separation lines, and merged cells, and other complex structures.

[0029] Inputting prompt text containing table structure extraction instructions into the visual large model is a key step in achieving accurate table analysis. The prompt text contains three main types of instructions: table row and column identification instructions, cell merge detection instructions, and content extraction instructions. Table row and column identification instructions, such as "identify the number of rows and columns in the table" and "determine the row and column boundaries of the table", are used to guide the model to analyze the basic grid structure of the table; cell merge detection instructions, such as "detect and mark all merged cells" and "identify cells that span rows or columns", are used to discover irregular table structures; content extraction instructions, such as "extract the text content in each cell" and "identify numbers and text in cells", are used to obtain the actual information within the cell.

[0030] A specific example of prompt text is: "Please analyze this table image, identify the number of rows and columns in the table, mark the location and range of all merged cells, and extract the text content in each cell. Output the analysis results in JSON format, including the number of rows and columns in the table, the coordinate position of each cell, merge information, and text content." Such prompt text not only indicates the task to be performed but also specifies the output format, effectively guiding the model to understand the table structure and generate the required analysis results.

[0031] When the visual model analyzes structural features based on prompt text, it sequentially performs row and column recognition, merged cell detection, and content extraction. During the row and column recognition phase, the model determines the basic grid structure of the table by detecting horizontal and vertical lines or cell boundary spacing. The model outputs the total number of rows and columns in the table. For example, for a table with 4 rows and 5 columns, the recognition result will include information such as the number of rows = 4 and the number of columns = 5.

[0032] During the cell merge detection phase, the model identifies merged cells by analyzing the continuity and break points of cell boundaries. For example, when a cell is detected spanning multiple grid positions and has no internal dividers, the model marks it as a merged cell. For each merged cell detected, the model records its starting row and column index and its ending row and column index. For example, a merged cell spanning from row 1, column 2 to row 2, column 3 would be recorded as {rowSpan: 2, colSpan: 2, startRow: 0, startCol: 1, endRow: 1, endCol: 2}.

[0033] During the content extraction phase, the model uses optical character recognition technology or its own text recognition capabilities to extract the text or numerical content within each cell. Even cells containing complex elements such as charts and symbols can be recognized and appropriately described. For example, if a cell contains the text "Total Sales," the model will record this text as a cell content attribute.

[0034] The visual large model finally outputs a table structure description result containing cell position information, merging information, and content information. This result is usually in a structured format such as JSON or XML, facilitating subsequent processing and application. The output result contains the following main information: the number of rows and columns of the table; the coordinate position of each cell, usually represented by row and column indices; the merging information of the cell, including whether it is a merged cell and the range of merging; and the content information of the cell, including text, numbers, or other content type descriptions.

[0035] In this way, the system can accurately extract structured information from table images, realizing the digitization and intelligent understanding of table content. Experiments show that when processing various complex tables using the above method, the model can achieve a structure recognition accuracy of over 95% and a content extraction accuracy of over 90%, greatly improving the efficiency and accuracy of table data processing. This table structure analysis method based on visual large models has wide application prospects in financial report analysis, academic paper data extraction, business intelligence, and other fields.

[0036] In an optional implementation, a synthetic table dataset is constructed, the visual large model is fine-tuned based on the synthetic table dataset, and low-rank adaptation technology is used to realize efficient updating of model parameters, including: The synthetic table dataset is constructed, and the synthetic table dataset contains table samples with multiple levels of table headers, merged cells, and nested structures. Random numerical content of the table samples is generated using structured rules. The visual large model is fine-tuned based on the synthetic table dataset. The table samples are input into the visual large model, and the structure recognition loss of the visual large model is calculated based on the true structure information of the table samples. The parameters of the visual large model are updated using low-rank adaptation technology. The number of trainable parameters is reduced through low-rank matrix decomposition methods to realize efficient updating of the parameters of the visual large model.

[0037] In the synthetic table dataset construction stage, the system generates table samples containing multiple levels of table headers, merged cells, and nested structures in a programmatic manner. Specifically, the system first defines a basic structure template for the table, including the number of rows, the number of columns, the level of table headers, and the position information of merged cells. For example, for a financial report template, a 3-level table header can be set, with the main table header containing "Balance Sheet", the second-level table header containing "Asset Class" and "Liability Class", and the third-level table header containing specific subjects such as "Current Assets", "Non-current Assets", "Current Liabilities", "Non-current Liabilities", etc. In the table body part, the merged cell area is set, such as merging "Cash", "Accounts Receivable", "Inventory", etc. under "Current Assets" into one cell to display the total data.

[0038] The table content generation adopts structured rules to set corresponding value generation rules according to different types of tables (financial statements, sales data, product specifications, etc.). Taking financial statements as an example, the system-generated values need to satisfy the accounting identity, such as "assets = liabilities + owner's equity". In specific implementation, the system randomly generates basic values and then calculates related items according to the preset rules. For example, the system automatically calculates the total of current assets as 998.3 million yuan based on the randomly generated values of each item of current assets: cash of 523.6 million yuan, accounts receivable of 346.2 million yuan, and inventory of 128.5 million yuan. For time series data, the system also simulates the time variation trend of the data to ensure the logical reasonableness of the data.

[0039] To enhance the diversity of the data set, the system also randomly adjusts the visual presentation of the table, including font size (range 9-14pt), cell inner margin (range 2-5 pixels), table line thickness (range 1-3 pixels), and background color (light colors such as #F5F5F5, #E8F4FD, etc.). At the same time, to simulate the image quality problems of tables in actual scenarios, noise, blur, and rotation (within ±5 degrees) are added to disturb the model and improve its robustness. The final generated synthetic table data set contains 10,000 samples, covering table types in multiple fields such as finance, sales, inventory, and human resources.

[0040] In the fine-tuning training phase, the system inputs the synthesized table samples into the visual large model to optimize the model's ability to recognize complex table structures through specific training strategies. Specifically, the system inputs the synthesized table images as input, extracts features through the encoder of the visual large model, and then predicts the structure information and content of the table through the decoder.

[0041] During training, the system designs a loss function for table structure recognition, including cell boundary detection loss, cell merging relationship loss, and table header hierarchical relationship loss. The cell boundary detection loss measures the difference between the predicted cell boundary and the true boundary using pixel-level comparison; the cell merging relationship loss evaluates the accuracy of the model's identification of merged cells by calculating the overlap between each predicted merged cell and the true merged cell; and the table header hierarchical relationship loss measures the accuracy of the model's understanding of the table header hierarchy by comparing the predicted table header hierarchy with the true hierarchy.

[0042] During the training implementation, the system adopts a batch processing mode, each batch contains 16 table samples, uses an adaptive learning rate strategy, the initial learning rate is set to 0.0001, and is attenuated to 0.9 times of the original value every 10 training periods. To prevent overfitting, a regularization term with a weight decay of 0.01 is added. The model starts to converge after 3 training periods, and a total of 10 periods are trained to achieve the best effect.

[0043] The efficient updating of model parameters using low-rank adaptation technology is the core innovation of the method. Traditional fine-tuning methods require updating all model parameters, which is computationally and storage-intensive. In this embodiment, low-rank adaptation technology is used to reduce the number of trainable parameters through low-rank matrix decomposition. The specific implementation is that the weight matrix W of the visual large model (assuming the dimension is d x k) is not directly updated, but two low-rank matrices A (dimension d x r) and B (dimension r x k) are introduced, where r is much smaller than d and k (in this embodiment, r is set to 8% of min(d, k)), and the update of W is approximated by the product of A and B.

[0044] In actual implementation, the original weight matrix W remains in a frozen state, and only the low-rank matrices A and B are trained. W + A x B is used as the actual weight during model forward propagation. This method reduces the number of trainable parameters from d x k to r x (d + k), significantly reducing the computational and storage overhead. For example, for a weight matrix with dimensions 1024 x 1024, the traditional method requires updating approximately 1 million parameters, while using low-rank adaptation (r = 82) only requires updating approximately 16.8 thousand parameters, a reduction of approximately 83%.

[0045] Experimental results show that the visual large model trained by this method achieves significant improvement in complex table recognition tasks. On the test set, the model's recognition accuracy for multi-level table headers improved from 78.2% to 92.5%, for merged cells from 81.5% to 94.3%, and for nested table structures from 75.3% to 90.8%. At the same time, the low-rank adaptation technology shortens the training time of model fine-tuning by 65% and reduces the computational resources required for training by 70%, demonstrating good efficiency and practicality.

[0046] In an alternative embodiment, the structural description information output by the visual large model is input into a language large model, and the language large model performs semantic extraction, formula derivation and verification to generate a standardized table structure output, which includes: The structural description information output by the visual large model is input into a language large model, and the structural description information contains cell position information, cell merging information, and cell content information of the table, and the language large model performs semantic understanding on the structural description information. Semantic extraction and formula deduction are performed based on the language big model. The language big model identifies the numerical relationships and calculation formulas in the table, verifies the correctness of the numerical relationships and the calculation formulas, ensures the logical consistency of the table data, and generates a standardized table structure output by the language big model.

[0047] The structural description information output by the large visual model is input into the large language model, which then performs semantic extraction, formula deduction, and verification to generate a standardized table structure output. This technical solution addresses the limitations of traditional table recognition systems in processing complex table structures and verifying data relationships.

[0048] The visual model analyzes table images and extracts structural information about the table, including cell location, cell merge information, and cell content information. Location information is expressed as coordinates, such as the top-left corner (x1, y1) and bottom-right corner (x2, y2). Merge information records cells that span rows or columns, such as "A1:B2" for the merged cells A1 through B2. Content information includes the text, numbers, or symbols within the cell.

[0049] These structural description information are organized into a structured JSON format, where each cell is represented as an object, including its location, merge status, and content properties. For example, a simple structural description can be represented as: "cell id ": "A1", "position": {"x1": 10, "y1": 15, "x2": 60, "y2": 45}, "merged": false, "content": "Sales". This structured representation facilitates subsequent processing by large language models.

[0050] After receiving the structural description information, the language model first performs semantic understanding. Through contextual analysis, the model identifies semantic elements of the table, such as the header, data area, subtotal row, and total row. For example, it identifies the first row as the header and the last row as the total row. The model also understands the semantics of the cell content and distinguishes different types of data, such as titles, categories, values, and percentages.

[0051] During the semantic extraction phase, the language model analyzes the semantic relationships within the table. For financial statements, the model identifies relationships between financial concepts such as "revenue," "cost," and "profit." For statistical tables, the model identifies summary relationships such as "subtotal" and "total." By analyzing the relative position and content of cells, the model constructs a semantic structure diagram for the table, grouping cells into logically related blocks.

[0052] The formula derivation stage is the core of the technical solution. The language large model analyzes the numerical relationship in the table and identifies the implicit calculation formula. For example, for a row containing "100", "200", and "300" numerical cells, if the fourth cell content is "600", the model infers that it is a summation relationship. The model supports basic operations such as addition, subtraction, multiplication, and division, as well as compound operations such as summation, average, and percentage.

[0053] For the identified numerical relationship, the language large model performs calculation verification. Taking the sales data table as an example, if a row contains "quarterly sales" of 300 million yuan, and the sales of the three months above it are 100 million yuan, 120 million yuan, and 80 million yuan respectively, the model calculates 100+120+80=300, verifying the consistency of the data. When inconsistencies are found, the model marks errors, such as "quarterly sales calculation error, should be 300 million yuan".

[0054] The verification process not only involves horizontal calculation relationships, but also includes vertical summary relationships, such as for a table containing sales data of multiple departments, the model verifies whether the sum of the sales of each department is equal to the total sales. The model also checks whether the percentage data is correct, such as one product accounting for 30% and another product accounting for 70%, the total should be 100%.

[0055] Based on semantic understanding and data verification, the language large model generates standardized table structure output, including: standardized table structure, such as row number, column number, and cell merging status; standardized representation of cell content, unified numerical format (such as currency, percentage, date); and calculation relationship in the table, with clear formula to represent the dependency between cells.

[0056] A specific example is processing a quarterly sales report. The visual large model extracts the table from the image and identifies the sales data of multiple products in different months, as well as monthly subtotals and product totals. The language large model analyzes and finds that the 5th column should be the sum of columns 2-4, and the 6th row should be the sum of rows 2-5. During the verification process, the model finds that the quarterly total sales of product C is 325 million yuan, but the sum of monthly sales is 330 million yuan (120+90+120), marking this inconsistency. In the final output, the model not only provides a standardized table structure, but also marks the discovered error and provides the correct value.

[0057] The advantage of this technical solution is that it combines the image understanding ability of the visual large model and the semantic analysis ability of the language large model, achieving deep understanding and verification of table data. Through semantic extraction, the table structure and content relationship are identified, and the data consistency is verified through formula derivation. Finally, a high-quality standardized table output is generated, greatly improving the accuracy and usability of table recognition.

[0058] In an optional implementation, the trained visual large model is used as a first teacher model to train a lightweight visual model, and the trained language large model is used as a second teacher model to train a lightweight language model, which comprises: The trained visual large model is used as a first teacher model to construct a knowledge distillation training framework of the first teacher model and the lightweight visual model, and the table structure features output by the first teacher model are used to guide the parameter learning of the lightweight visual model. The trained language large model is used as a second teacher model to construct a knowledge distillation training framework of the second teacher model and the lightweight language model, and the semantic understanding features output by the second teacher model are used to guide the parameter learning of the lightweight language model.

[0059] A pre-trained visual large model, such as a visual model based on a Transformer architecture, is prepared, which has 24 Transformer layers, each containing 16 attention heads, a hidden layer dimension of 1024, and a total parameter quantity of about 300M. As a comparison, a target lightweight visual model adopts 6 Transformer layers, each containing 8 attention heads, a hidden layer dimension of 512, and a total parameter quantity of about 25M. When constructing the knowledge distillation framework, the same image training dataset is used, which contains 1 million labeled images and covers various scenes, objects, and visual elements. The input images are uniformly adjusted to 224x224 size and standardized.

[0060] In the knowledge distillation process, the same batch of image data is input into the teacher model and the student model. The table structure features extracted by the teacher model include the following aspects: boundary box positioning of the table structure in the image, extraction of the spatial position relationship information of the table cells, and visual features of the table content. Specifically, the teacher model outputs a table boundary detection feature vector (dimension 1024), a cell position feature vector (dimension 1024x number of cells), and a table structure relationship feature vector (dimension 1024). The student model also outputs feature vectors of the same dimension, but the dimension is 512.

[0061] In the loss function design, a feature alignment loss and a task-specific loss are combined. The feature alignment loss calculates the difference between the features output by the teacher model and the student model, and minimizes this difference to enable the student model to learn the knowledge representation ability of the teacher model. At the same time, task-specific losses for table detection and recognition are introduced to ensure the performance of the student model on specific tasks. The Adam optimizer is used in the training process, the initial learning rate is set to 0.0001, and the cosine annealing strategy is used to adjust the learning rate. The batch size is 32, and the training is performed for 50 cycles. During the training process, the loss value is recorded every 10 batches, and the model performance is evaluated on the validation set at the end of each cycle.

[0062] In practical applications, the lightweight visual model achieves an accuracy of 94.5% on the table detection task, and a table structure recognition accuracy of 92.3%. Compared with the original visual large model (96.2% and 94.8% respectively), there is only a small decline, while the inference speed is increased by 8.5 times and the memory occupation is reduced by 11 times.

[0063] The specific process of training a lightweight language model using a trained language large model as a second teacher model is as follows: a pre-trained language large model is selected, which has a 12-layer Transformer structure, an attention head number of 16, a hidden layer dimension of 768, and a total parameter quantity of about 120M. The target lightweight language model adopts a 4-layer Transformer structure, an attention head number of 8, a hidden layer dimension of 384, and a total parameter quantity of about 15M. The training data set contains 500,000 table-related text descriptions and question and answer pairs, which cover different fields of table content explanation, query, and analysis scenarios.

[0064] In the knowledge distillation training framework, for each text input, standardization processing is first performed, including tokenization, case unification, etc. The maximum length of the text sequence is limited to 512 tokens. The same text input is sent to the teacher model and the student model for processing. The semantic understanding features output by the teacher model mainly include: context representation vector (dimension 768 x sequence length), semantic relationship attention map (dimension 16 x sequence length x sequence length), and understanding vector for table content (dimension 768). The student model outputs corresponding features, but with half the dimension.

[0065] In the knowledge distillation process, the student model learns the knowledge of the teacher model through multiple loss functions, including intermediate layer feature alignment loss, which aligns the output features of each layer of the student model and the teacher model to enable the student model to learn the internal representation ability of the teacher model; attention map alignment loss, which aligns the attention distribution of the two models to enable the student model to learn the focus points of the teacher model; and output layer prediction loss, which ensures the performance of the student model on specific prediction tasks. The AdamW optimizer is used for training with an initial learning rate of 0.0002 and a linear learning rate decay strategy. The batch size is 64 and the training is performed for 30 cycles. The performance of the validation set is evaluated every 500 steps and the best checkpoint is saved.

[0066] Through the above knowledge distillation training, the lightweight language model obtained on the table content understanding task achieves an accuracy of 88.7%, and the table question and answer task accuracy is 85.9%. Compared with the original language large model (91.3% and 89.2% respectively), the performance loss is controlled within a reasonable range, while the inference speed is increased by 6.3 times and the memory occupation is reduced by 7.8 times.

[0067] After the training of the two lightweight models is completed, they can be integrated in the same framework for end-to-end table recognition and understanding tasks. The integration test shows that on a resource-limited mobile device (configured with 4GB RAM and a quad-core processor), the lightweight model combination can complete the detection, structure recognition and content understanding of a complex table in 2 seconds, while the original large model combination takes more than 15 seconds and cannot run on some low-end devices. This proves that the lightweight models obtained through knowledge distillation significantly improve the feasibility of practical application while maintaining high performance.

[0068] In an optional implementation, the lightweight visual model and the lightweight language model are deployed on an edge device, and offline processing of the table through the lightweight visual model and the lightweight language model includes: The lightweight visual model and the lightweight language model are deployed on an edge device, and the parameter amount of the lightweight visual model and the lightweight language model is controlled within the computing resource limit of the edge device; The lightweight visual model is called on the edge device to perform offline structure recognition of the table, and the lightweight visual model outputs structure description information of the table, which is directly input to the lightweight language model; The lightweight language model is called on the edge device to perform offline semantic processing on the structure description information, and the lightweight language model generates standardized table output results, realizing end-to-end offline processing of the table.

[0069] The lightweight visual model and the lightweight language model are deployed on an edge device, which can be a smartphone, a tablet computer, a smart camera or other terminal devices with certain computing power but limited resources. These devices usually have limited computing power, storage space and energy supply. In order to adapt to the computing resource limit of the edge device, the parameter amount of the lightweight visual model and the lightweight language model needs to be controlled. In practical applications, the parameter amount of the lightweight visual model is controlled within 10MB, and the parameter amount of the lightweight language model is controlled within 50MB. For example, model pruning technology can be used to prune the original visual model, reducing the number of channels of the convolutional layer from 512 to 128, thereby reducing the model parameter amount from 60MB to 8MB. For the language model, knowledge distillation technology can be used to distill the original large language model (e.g. 12-layer Transformer structure, parameter amount of 300MB) into a small model with 4-layer Transformer structure, with a parameter amount of only 40MB.

[0070] After deployment, the lightweight visual model is invoked on the edge device for offline structure recognition of the table. The specific process includes: obtaining the image containing the table through the camera of the edge device or from local storage, preprocessing the image, including scaling to a uniform size (such as 640x480 pixels), standardization, etc.; inputting the preprocessed image into the lightweight visual model, which uses the Faster R-CNN architecture and includes three main components: feature extraction network, region proposal network, and table structure recognition network. The feature extraction network uses depth separable convolution instead of standard convolution to reduce computational complexity; the region proposal network is responsible for identifying table regions; the table structure recognition network further analyzes the internal structure of the table to identify cell boundaries, row and column distribution, etc.

[0071] The lightweight visual model outputs the structure description information of the table, which represents the layout and content of the table in a structured format. The structure description information includes: table boundary coordinates (such as left upper corner coordinates [50, 100] and right lower corner coordinates [550, 400]); number of rows and columns (such as 5 rows and 4 columns); position information of each cell (such as the left upper corner coordinates [50, 100] and right lower corner coordinates [175, 150] of cell [0, 0]); cell merging information (such as the merging of cells [0, 0] and [0, 1]); and text content in the cell (such as the text "Product Name" in cell [0, 0]). These structure description information is organized in JSON format and directly input into the lightweight language model.

[0072] The structure description information is directly input into the lightweight language model for offline semantic processing. The lightweight language model is based on a simplified version of the Transformer architecture and includes self-attention mechanisms and feedforward neural networks. Compared to the original Transformer model, this lightweight model reduces the number of attention heads (from 8 to 4) and reduces the intermediate layer dimension (from 2048 to 512). In addition, INT8 quantization technology is used to quantize model parameters from FP32 (32-bit floating point numbers) to INT8 (8-bit integers), further reducing model size and computational complexity.

[0073] The lightweight language model performs semantic understanding and processing on the structure description information, including: parsing the table layout and content in the structure description information; analyzing the semantic structure of the table to identify components such as table headers, data areas, and table notes; understanding the logical relationships between cells, such as the correspondence between topics and attributes, categories and numerical values; handling special cases such as cross-row and cross-column cells, empty cells, etc.; and standardizing table content, such as unifying date, currency, and unit of measurement formats.

[0074] After semantic processing, the lightweight language model generates standardized table output results, which are usually in common formats such as CSV, Excel, JSON, or HTML, facilitating subsequent application processing or user viewing. For example, for a product price table, there are various formats of dates and price representations in the original image (such as "January 5, 2023" and "¥1,234.56"), and after processing, the dates in the output results are unified as "2023-01-05" format, and the prices are unified as "1234.56" numerical format.

[0075] In actual application cases, the method processes a sales data table containing 10 rows and 7 columns. The recognition time of the lightweight visual model on the edge device (with a 2GHz quad-core processor and 4GB of memory) is about 0.8 seconds, and the accuracy rate reaches 95%; the processing time of the lightweight language model is about 0.5 seconds, and the semantic understanding accuracy rate reaches 93%. The entire end-to-end offline processing process is completed within 1.5 seconds and does not require networking, protecting data privacy and security. Through the above technical implementation, efficient offline processing of tables on edge devices is realized.

[0076] The method further comprises: When preprocessing the manufacturing industry PDF form, the PDF document is scanned or rasterized at a resolution of 300 dpi to generate a 24-bit true color image. A deep convolutional neural network is used to detect the table area of the page, with the network input being the entire page image and the output being the bounding box coordinates of the table area. The detected table area is projected and transformed to correct the image tilt angle. Image enhancement uses an adaptive histogram equalization method to improve contrast and a USM sharpening algorithm to enhance edge details. The image is denoised using a 5x5 Gaussian filter kernel to filter out salt and pepper noise generated during scanning. The border line removal uses a morphological opening operation with a structure element size of 3x3 pixels and iterates 2 times to complete border line extraction and elimination.

[0077] The visual large model recognition stage uses a visual language model with 72 billion parameters, which includes 48 layers of Transformer encoder. A structured prompt template is constructed: "Analyze the table structure and output the cell position and content. Each cell contains: row number (starting from 0), column number (starting from 0), whether it is merged (merge range), and cell text." Based on the prompt text and preprocessed table image, the model extracts visual features through a multi-head attention mechanism to generate a table structure description.

[0078] The fine-tuning optimization stage constructs a synthetic table dataset containing 5000 samples. The table samples contain 2-4 level header structures, with a cell merging rate of 15%-30%, and a numerical cell proportion of 40%-60%. The low-rank adaptation technique is used to fine-tune the visual large model, decomposing the fully connected layer into the product of low-rank matrices to reduce the number of trainable parameters. The learning rate is 0.0001, the batch size is 4, and the number of training rounds is 10. At the same time, the prompt template is optimized to increase structural constraints and numerical verification instructions.

[0079] The language processing stage uses an instruction-based language model with 72 billion parameters. The model receives the structural description text output by the visual model, extracts the header level relationship and unit information. The numerical cells are subjected to arithmetic verification, such as in an investment estimation table, the total value of the equipment purchase expense column is 12.5 million yuan, which is checked by summing the sub-amounts in the column, and it is found that the actual total should be 12.8 million yuan, and the model automatically corrects the recognition error. The final output is a standardized table structure in JSON format, containing complete cell attributes and verified numerical values.

[0080] In the lightweight deployment stage, the 72 billion parameter visual and language models are distilled to 1 billion parameter models respectively. The teacher-student architecture is used for knowledge transfer, and the student model is reduced to 12 layers. The model weights are quantized to 8 bits, and the low-rank decomposition is used to reduce the computational complexity. The average processing time of a single table for the distilled lightweight model is 0.8 seconds, which can achieve real-time processing on an industrial computer with a main frequency of 2.5 GHz.

[0081] In the practical application of the manufacturing investment estimation table, the table contains 4-level headers such as investment categories and cost centers, and 30% of the cells have merging relationships. The system processes the table, and the visual model accurately identifies the header level structure and merging cell information, and the language model corrects 3 numerical calculation errors. The final reconstructed table structure retains the original layout characteristics, and the numerical verification accuracy is above 95%. The experimental results show that the system can effectively process complex tables in the manufacturing industry and realize end-to-end recognition from image input to structured data output.

[0082] The method further comprises: The intelligent multi-source heterogeneous data collection and storage platform is constructed. For structured and semi-structured data, the system flexibly configures the connection information of the data source through the data source configuration management module, and dynamically allocates reading and writing permissions. In practical application cases, the system can process data sources including device parameter tables, ship operation logs, maintenance records, etc. After configuration, the system performs connectivity test, and the test pass rate reaches 99.5%, ensuring the stability of the data channel. The system automatically generates the data source mapping relationship between the source and the target, for example, mapping the pressure sensor data of a certain type of submarine from the original database to the corresponding field in the target data warehouse.

[0083] Based on a distributed computing engine (such as Spark), the system adopts a dual-track parallel working mode of script mode and visual canvas mode. Through the visual interface, technical personnel can intuitively configure the structure mapping relationship between the source and the target, and dynamically adjust key parameters such as memory allocation, concurrency degree, and data flow rate according to the data magnitude. Actual tests show that for 10GB level ship equipment data, the system can complete the data extraction and cleaning task within 15 minutes, and maintain 99.9% data integrity. Through built-in filtering, deduplication, conversion and other logical components, the system builds an efficient data extraction and cleaning task, realizes the accurate migration and high-quality precipitation of data from the source to the target storage layer, and finally extracts to the distributed file sharing system.

[0084] For unstructured data, it is uniformly extracted and stored to the distributed file sharing system through the data collection platform, including submarine technical documents, design drawings, maintenance manuals, operation videos and other diversified materials. Combined with natural language processing and computer vision technology, the system constructs a multi-modal data analysis engine to realize intelligent identification and structured conversion of complex data such as text, image, audio and video. For example, for the PDF document of the submarine maintenance manual, the system can automatically identify the document structure, extract key parameters, operation steps and fault solution, and convert it into a structured data table, with an accuracy of 92.3%. For submarine equipment image data, the system can identify device model, key component position and connection relationship, and generate a standardized device information table, with an identification accuracy of 89.7%. Through these processes, unstructured data is included in the unified data governance system, providing high-value data for subsequent analysis and application.

[0085] The system builds a multi-modal intelligent perception engine to achieve accurate identification and preprocessing of industrial data. The intelligent perception engine can handle multiple data types, including common documents, tables, plain text formats, and other formats. In practical applications, the system has processed over 500,000 multi-modal documents in the submarine field, covering text, images, and videos. The file preprocessing module extracts key information, optimizes table formats, cleanses text, enhances and denoises images, and performs multi-modal recognition for different types of files. For example, for submarine sonar equipment images containing noise, the system uses an adaptive filtering algorithm for noise reduction, improving image quality and making device features more clear and identifiable, with recognition accuracy improving from 76% to 92%. The large model processing module provides multiple large language and multi-modal large model options to adapt to different needs. The system builds a dedicated Prompt template library based on the characteristics of the submarine industry, containing over 200 field-specific prompt templates for accurately extracting submarine equipment parameters, performance indicators, and maintenance information.

[0086] In the data insight and preprocessing stage, the system analyzes the distribution of data samples and performs multi-dimensional analysis on submarine equipment data to find numerical distribution patterns and abnormal points. The system automatically cleanses abnormal characters, such as removing special symbols and garbled characters mixed in the text, improving data quality. It also removes duplicate information, such as merging duplicate content of a certain type of submarine technical parameters in multiple documents, reducing redundancy. The system also unifies the unit format of numerical information, such as converting different document pressure units "MPa", "bar", and "kg / cm²" to the standard unit "MPa" for easy comparison. Through the selection and processing of data, the system filters out invalid information, such as removing non-core content such as document headers and footers, watermarks, etc. Sensitive information is encrypted to ensure data security. Finally, the system disassembles and maps equipment indicators, breaking down complex submarine performance indicators into basic parameters and establishing mapping relationships between parameters.

[0087] Output structured data results such as equipment information detail table, equipment information detail table, project library information detail table, and equipment indicator detail table. These detail tables contain standardized submarine equipment parameters, performance indicators, maintenance records, and operation manuals, providing a foundation for subsequent data analysis and decision support.

[0088] A deep learning model is constructed by fusing a transformer architecture and an expert hybrid mechanism. In the model selection stage, the system optimizes open-source Base models such as Qwen and LLama. In the cold start stage, the system selects thousands of high-quality manually annotated submarine field samples for fine-tuning, including submarine equipment operation manuals, fault diagnosis cases, and maintenance records. Through a small amount of prompting combined with long-chain thinking reasoning technology, the model generates answers containing reflection verification, and collects and processes the output into high-quality fine-tuning data. In the reasoning-oriented reinforcement learning stage, the system designs a language consistency reward mechanism to calculate the reward value based on the proportion of target language words in chain thinking reasoning, thereby improving language consistency. Actual tests show that the accuracy of the model in using submarine professional terms has increased from 78% of the base model to 92%.

[0089] In the rejection sampling and supervised fine-tuning stage, the reinforcement learning model is converged for rejection sampling to generate new samples. The system expands the supervised fine-tuning dataset to cover multiple fields such as submarine design, construction, operation, and maintenance, thereby improving the model's general ability. Through full-scene-oriented reinforcement learning, the system implements a two-stage reinforcement learning strategy, which aims to cover all types of submarine field tasks. The system uses different reward signals and prompt distributions for different tasks, such as strengthening accuracy for fault diagnosis tasks and safety and step integrity for operation guidance tasks. In the efficient fine-tuning stage of vertical fields, the system collects specific field data such as submarine power systems and acoustic systems, and uses parameter-efficient fine-tuning technology to optimize the model's performance in specific professional fields. Actual test results show that the accuracy of the optimized model in answering submarine professional questions reaches 93.7%, which is 17 percentage points higher than that of the base model.

[0090] In a second aspect of the embodiment of the present application, a multi-modal data preprocessing and fusion system based on artificial intelligence is provided, comprising: A first unit is configured to convert an input PDF document into a table image, and preprocess the table image, including table region detection, direction correction, image enhancement, noise reduction processing, and frame line removal, to obtain a preprocessed table image. A second unit is configured to perform structure analysis on the preprocessed table image using a visual large model, input a prompt text containing table structure extraction instructions into the visual large model, the prompt text being used to guide the visual large model to output structure description information of the table, the structure description information including row numbers, column numbers, merging information and contents of each cell in the table, and realizing zero-sample table structure understanding based on the prompt text. The third unit is configured to construct a synthetic table dataset, fine-tune the visual large model based on the synthetic table dataset, and realize efficient updating of model parameters by using a low-rank adaptation technology; and input structural description information output by the visual large model into a language large model, perform semantic extraction, formula deduction and verification by the language large model, and generate a standardized table structure output. The fourth unit is configured to train a lightweight visual model by taking the trained visual large model as a first teacher model, train a lightweight language model by taking the trained language large model as a second teacher model, deploy the lightweight visual model and the lightweight language model on an edge device, and realize offline processing of a table by using the lightweight visual model and the lightweight language model.

[0091] In a third aspect, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0092] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0093] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0094] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An artificial intelligence-based multi-modal data preprocessing and fusion technology, characterized in that, The method comprises the following steps: Converting an input PDF document into a table image, preprocessing the table image, including table region detection, direction correction, image enhancement, noise reduction processing and border line removal, to obtain a preprocessed table image; Using a visual large model to analyze the structure of the preprocessed table image, inputting a prompt text containing table structure extraction instructions into the visual large model, the prompt text being used to guide the visual large model to output structure description information of the table, the structure description information including row number, column number, merging information and content of each cell in the table, and realizing zero-shot table structure understanding based on the prompt text; Constructing a synthetic table dataset, fine-tuning the visual large model based on the synthetic table dataset, and realizing efficient updating of model parameters by using low-rank adaptation technology; Inputting the structure description information output by the visual large model into a language large model, executing semantic extraction, formula deduction and verification by the language large model, and generating standardized table structure output; Using the trained visual large model as a first teacher model to train a lightweight visual model, using the trained language large model as a second teacher model to train a lightweight language model, deploying the lightweight visual model and the lightweight language model on an edge device, and realizing offline processing of the table by the lightweight visual model and the lightweight language model.

2. The method of claim 1, wherein, Using a visual large model to analyze the structure of the preprocessed table image, inputting a prompt text containing table structure extraction instructions into the visual large model comprises: Using a visual large model to analyze the structure of the preprocessed table image, the visual large model extracting structural features of the table from the table image through a multi-head attention mechanism; Inputting a prompt text containing table structure extraction instructions into the visual large model, the prompt text containing table row and column recognition instructions, cell merging detection instructions and content extraction instructions, the prompt text being used to guide the visual large model to understand the table structure; Based on the prompt text, the visual large model analyzes the structural features, and outputs table structure description results containing cell position information, merging information and content information.

3. The method of claim 1, wherein, Constructing a synthetic table dataset, fine-tuning the visual large model based on the synthetic table dataset, and realizing efficient updating of model parameters by using low-rank adaptation technology comprises: Constructing a synthetic table dataset, the synthetic table dataset containing table samples with multi-level headers, merged cells and nested structures, and generating random numerical content of the table samples by using structured rules; Fine-tuning the visual large model based on the synthetic table dataset, inputting the table samples into the visual large model, and calculating structure recognition loss of the visual large model based on true structure information of the table samples; Updating parameters of the visual large model by using low-rank adaptation technology, reducing the number of trainable parameters by using low-rank matrix decomposition method, and realizing efficient updating of parameters of the visual large model.

4. The method of claim 1, wherein, Inputting the structural description information output by the visual large model into a language large model, performing semantic extraction, formula deduction and verification by the language large model, and generating a standardized table structure output including: Inputting the structural description information output by the visual large model into a language large model, the structural description information including cell position information, cell merging information and cell content information of the table, and performing semantic understanding on the structural description information by the language large model; Based on the semantic extraction and formula deduction performed by the language large model, the language large model identifies the numerical relationship and calculation formula in the table, verifies the correctness of the numerical relationship and the calculation formula, and ensures the logical consistency of the table data, and generates a standardized table structure output by the language large model.

5. The method of claim 1, wherein, Training a lightweight visual model by taking the trained visual large model as a first teacher model, and training a lightweight language model by taking the trained language large model as a second teacher model includes: Taking the trained visual large model as a first teacher model, constructing a knowledge distillation training framework of the first teacher model and the lightweight visual model, and guiding the parameter learning of the lightweight visual model through the table structure features output by the first teacher model; Taking the trained language large model as a second teacher model, constructing a knowledge distillation training framework of the second teacher model and the lightweight language model, and guiding the parameter learning of the lightweight language model through the semantic understanding features output by the second teacher model.

6. The method of claim 1, wherein, Deploying the lightweight visual model and the lightweight language model on an edge device, and realizing offline processing of the table through the lightweight visual model and the lightweight language model includes: Deploying the lightweight visual model and the lightweight language model on an edge device, and controlling the parameter amount of the lightweight visual model and the lightweight language model within the computing resource limit of the edge device; Calling the lightweight visual model on the edge device for offline structure recognition of the table, outputting the structural description information of the table by the lightweight visual model, and directly inputting the structural description information into the lightweight language model; Calling the lightweight language model on the edge device for offline semantic processing of the structural description information, generating a standardized table output result by the lightweight language model, and realizing end-to-end offline processing of the table.

7. The multi-modal data pre-processing and fusion system based on artificial intelligence, for implementing the method of any one of the preceding claims 1-6, characterized in that, It includes: The first unit is configured to convert an input PDF document into a table image, and pre-process the table image, including table region detection, direction correction, image enhancement, noise reduction processing and border line removal, to obtain a pre-processed table image; The second unit is configured to perform structure analysis on the pre-processed table image by using a visual large model, input a prompt text containing table structure extraction instructions into the visual large model, the prompt text being used to guide the visual large model to output structural description information of the table, the structural description information including row number, column number, merging information and content of each cell in the table, and realizing zero-shot table structure understanding based on the prompt text; The third unit is configured to construct a synthetic table dataset, fine-tune the visual large model based on the synthetic table dataset, and realize efficient updating of model parameters by using a low-rank adaptation technology. The structural description information output by the visual large model is input into a language large model, and the language large model is used to perform semantic extraction, formula deduction and verification to generate a standardized table structure output. The fourth unit is configured to train a lightweight visual model by taking the trained visual large model as a first teacher model, train a lightweight language model by taking the trained language large model as a second teacher model, and deploy the lightweight visual model and the lightweight language model on an edge device to realize offline processing of a table by using the lightweight visual model and the lightweight language model.

8. An electronic device, comprising: It comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, realize the method of any one of claims 1 to 6.

Images