Table processing method and electronic equipment

Abstract

The invention relates to a table processing method and electronic equipment. The method comprises the steps of obtaining a to-be-processed worksheet file; determining, through a large language model for table understanding, structured information of the worksheet file, the structured information comprising at least one of the following items of a worksheet for the worksheet file: information of a logical sub-table, row description information, or grouping information; and storing the structured information using a predefined format. In this way, the embodiment of the invention provides a method for understanding the structure of the table, and deep understanding of the internal semantic structure of the table is achieved.

Description

Technical Field

[0001] The present disclosure relates generally to the field of computers, and more particularly to a spreadsheet processing method and an electronic device. Background Art

[0002] In modern enterprise management, various worksheet files are widely used to record and manage business processes and serve as crucial data carriers. Many companies rely on worksheet files for process management, statistical data, and decision support. However, worksheet files offer a high degree of editing flexibility, and the layout structures of different worksheet files vary significantly in practical applications. This complexity not only makes them difficult for humans to read and understand, but also poses significant challenges to automated systems.

[0003] A Large Language Model (LLM) refers to a deep learning model trained on large amounts of text data. LLMs can understand and generate natural language, which is manifested in tasks such as answering questions, writing articles, translating, extracting summaries, and reasoning. With the continuous development of LLMs, question-answering systems based on natural language processing have been proposed for worksheet files. However, current question-answering systems mainly target worksheet files with standard tables with a regular structure and a single theme, and have low parsing and accuracy rates for highly complex worksheet files used in actual business. Therefore, the current question-answering systems have a limited scope of application and are not yet suitable for diverse enterprise scenarios. Summary of the Invention

[0004] According to example embodiments of the present disclosure, a form processing method, apparatus, electronic device, computer-readable storage medium, and computer program product are provided.

[0005] In a first aspect of the present disclosure, a table processing method is provided, comprising: obtaining a worksheet file to be processed; determining structured information of the worksheet file through a large language model for table understanding, the structured information including at least one of the following for a worksheet of the worksheet file: information of a logical sub-table, row description information, or grouping information; and storing the structured information using a predefined format.

[0006] In a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which instructions, when executed by the at least one processing unit, enable the electronic device to perform the method described in accordance with the first aspect of the present disclosure.

[0007] In a third aspect of the present disclosure, a table processing device is provided, including: an acquisition unit configured to acquire a worksheet file to be processed; a determination unit configured to determine structured information of the worksheet file through a large language model for table understanding, the structured information including at least one of the following for a worksheet of the worksheet file: information of a logical sub-table, row description information, or grouping information; and a storage unit configured to store the structured information using a predefined format.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which has machine-executable instructions stored thereon, and when the machine-executable instructions are executed by a device, the device can perform the method described according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, wherein the computer-executable instructions implement the method described according to the first aspect of the present disclosure when executed by a processor.

[0010] In a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processing circuit configured to execute the method described according to the first aspect of the present disclosure.

[0011] The purpose of providing the summary of the invention section is to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary of the invention section is not intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] Figure 1 A schematic diagram illustrating a system in which embodiments of the present disclosure can be applied;

[0014] Figure 2 A schematic flow chart illustrating a process for question answering according to some embodiments of the present disclosure is shown;

[0015] Figure 3 A schematic flow chart showing a method for table processing according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A block diagram illustrating an example apparatus according to some embodiments of the present disclosure; and

[0017] Figure 5A block diagram of an example computing device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It is understood that the drawings of the present disclosure and the embodiments are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] As mentioned above, there are great differences in the layout structure and the like of different worksheet files, and such complexity brings significant challenges to the parsing and utilization of table data by the question and answer system based on natural language processing. At present, the question and answer system for structured data has limited processing capacity and low accuracy in dealing with complex layout, multiple table headers, nested sub-tables and the like in actual business, and is difficult to adapt to diversified enterprise scenarios.

[0020] Retrieval Augmented Generation (RAG) technology is a model architecture combining "information retrieval" and "text generation", which can be used in question and answer systems or chat robots, for example. The use of RAG technology in question and answer systems brings new possibilities for the use of structured data. RAG can embed standardized table fragments into vector space and efficiently locate relevant information through semantic retrieval, showing certain advantages in the face of semi-structured data. However, when dealing with real business tables with complex structure and dense semantic association, RAG still has the shortcoming of limited reasoning ability and lack of dynamic processing flow.

[0021] In order to further improve the response capability of intelligent question and answer systems in complex table scenarios, in recent years, there have been schemes for introducing agents with planning and reasoning capabilities into the question and answer process. An agent is an autonomous system that can perceive its environment, make decisions, and take actions to achieve its goals. For example, an agent can be a software program, a robot, or a component of an intelligent system, etc. The agent can conduct multi-round thinking based on the retrieval results to determine whether to directly answer, perform step by step, or call external tools for calculation and analysis. The collaborative mode of RAG and agent is becoming an important trend for question and answer systems facing complex data scenarios, especially structured table data.

[0022] Embodiments of the present disclosure provide a table processing solution that can process spreadsheets. Spreadsheets are a type of electronic document used for data organization, analysis, and calculations. Spreadsheets display information in a table format consisting of rows and columns. The intersection of rows and columns can be called a "cell." For example, text, numbers, formulas, etc. can be entered in a cell.

[0023] The term "worksheet file" is used in this disclosure to refer to an object stored in a computer or memory, for example, a worksheet file can be a workbook. For example, in spreadsheet software, a workbook is a file, and a workbook can include one or more worksheets.

[0024] A worksheet is a page in a workbook used to enter, manage, and analyze data. Each worksheet can be composed of rows (horizontally) and columns (vertically), with cells forming the intersection of rows and columns. Each worksheet is a page composed of rows and columns, and can include one or more tables on a page.

[0025] A cell is the smallest unit of data in a worksheet and has a unique address. It can be represented by a row index and a column index. For example, column indexes can be represented by A, B, C, and so on, and row indexes can be represented by 1, 2, 3, and so on. Therefore, cell "A1" represents the cell in row 1 and column 1 of a worksheet.

[0026] Figure 1 FIG. 1 shows a schematic diagram of a system 100 in which an embodiment of the present disclosure can be applied. Figure 1 As shown, the system 100 may include a form processing system 110 and a form question answering system 120 .

[0027] As shown, the form processing system 110 may include a form parsing subsystem 112 and a form understanding subsystem 114. Optionally, the form processing system 110 may further include a form format conversion subsystem 116. Optionally, the form processing system 110 may be referred to as an Intelligent Document Processing (IDP) system. For example, the form processing system 110 may be deployed on a computer or a cloud server, and is designed to identify, classify, and process various documents (e.g., worksheet files of various sources and types) for an enterprise.

[0028] As illustrated, the table question answering system 120 can include a worktable RAG module 122 and / or a worktable agent module 124. Optionally, the table question answering system 120 can be referred to as an intelligent question answering system based on table understanding, or an enhanced RAG / agent question answering system based on table understanding. Exemplarily, the table question answering system 120 can be deployed on a computer or a server in the cloud, aiming to provide accurate and targeted answers for a user asking questions.

[0029] It can be understood that the table processing system 110 and the table question answering system 120 in the embodiments of the present disclosure can be deployed on the same or different electronic devices. For example, both can be deployed on different computers. For example, the table processing system 110 is deployed on a computer, and the table question answering system 120 is deployed on a server in the cloud, for example, a user can use the table question answering system 120 via an application or a web program on the computer. It can be understood that the methods or operations of the table processing system 110 and the table question answering system 120 discussed below can be executed by a processor or processing circuit in a computer or a server.

[0030] A user can provide or input a worktable file 101 to be processed, for example, the worktable file includes at least one worktable. The table parsing subsystem 112 can determine the information in the at least one worktable of the worktable file 101 through structured parsing.

[0031] In some embodiments, the worktable file 101 can be generated by (or using) a first application, that is, the first application is the document source of the worktable file 101. In the embodiments of the present disclosure, the system 100 can support processing of worktable files of multiple different document sources. For example, the multiple different document sources include different applications, for example, the worktable files of the multiple different document sources can have the same or different file name suffixes. For example, the worktable file 101 can be a native table type document of the first application, or can be a table block embedded in another type of document of the first application, etc., which is not limited in the present disclosure.

[0032] The worktable file 101 can include one worktable or multiple worktables, where one worktable can correspond to one page. The table parsing subsystem 112 can extract the information of each worktable by parsing. For the purpose of description, one worktable is exemplarily described.

[0033] The worksheet information extracted by the table parsing subsystem 112 may include information at the row granularity, information at the column granularity, information at the cell granularity, etc. For example, the extracted worksheet information may include or indicate some or all of the following information: whether row data is visible or hidden, whether column data is visible or hidden, filter information for column data, text, numbers, or formulas in cells, cell attribute information, expressions and calculation results of formulas in cells, cell comments and / or remarks, cell merging information and logical layout, charts embedded in the worksheet and mapping relationships between charts and data, etc.

[0034] Optionally, the filtering information of the column data may include whether a filter is set, filtering conditions and visibility rules, etc.

[0035] Optionally, the cell attribute information may include cell border attributes, such as whether there is a border, which side has a border, the line type of the border line, the width of the border line, the color of the border line, etc. Optionally, the cell attribute information may include cell color attributes, such as whether it is filled with a color, what color is filled with, etc. Optionally, the cell attribute information may include attributes of the text (such as text, numbers, or formulas, etc.) in the cell, such as font, font size, font color, alignment (such as center / left / right), whether it is bold, whether it is italic, whether it is underlined, and the line type of the underline, etc.

[0036] Optionally, the cell comment and / or remark content may include the author, the time of the comment and / or remark, the text information of the comment and / or remark (ie, the specific comment content), etc.

[0037] Optionally, the cell merging information and logical layout may include which cells are merged, attribute information of the merged cells (similar to the attribute information of the cells described above), and the like.

[0038] Optionally, the chart embedded in the worksheet and the mapping relationship between the chart and the data may include the object type of the embedded chart, the data based on which the embedded chart is generated, and the like.

[0039] In this way, the embodiments of the present disclosure provide a unified parsing method for multi-source, heterogeneous tabular data. This unified parsing framework is applicable to spreadsheet files from multiple sources and supports the extraction of various types of information: hidden cells, formulas, charts, filter conditions, annotations, and other structural information. This ensures that the semantics of the file are intact and the structure is not misinterpreted, providing an accurate foundation for subsequent semantic understanding of the table.

[0040] The table understanding subsystem 114 can perform semantic understanding of the worksheet based on the parsing results of the table parsing subsystem 112, thereby obtaining structured information about the worksheet. The table understanding subsystem 114 can include a large language model trained for table understanding. The large language model used for table understanding can be used to perform structural recognition on the worksheet to determine the structured information.

[0041] The structural information of the worksheet obtained by the table understanding subsystem 114 may include or indicate part or all of the following information: information of logical sub-tables, row description information, grouping information, and the like.

[0042] Optionally, row description information may include descriptive information about the logical sub-table in the worksheet. For example, table understanding subsystem 114 may determine a description row from the worksheet, such as a text row above, below, next to, or within a logical sub-table that serves a descriptive and explanatory purpose. For example, "Unit: 10,000 Yuan" in a logical sub-table is a descriptive description and constitutes row description information.

[0043] Optionally, the information of the logical sub-table may include whether the worksheet contains logical sub-tables, table information of each logical sub-table, etc. For example, the table information of the logical sub-table may include the table type, the area where the logical sub-table is located, the data area of ​​the logical sub-table, the header area of ​​the logical sub-table, the key columns of the logical sub-table, etc.

[0044] For example, the table understanding subsystem 114 may determine (or divide) multiple semantically independent logical sub-tables based on at least one of the following features: blank rows, blank columns, lines, merged cells, header alignment, cell formatting, etc. In this way, the area where each logical sub-table is located can be determined from the worksheet.

[0045] For example, for a logical sub-table (such as any one of the multiple logical sub-tables of a worksheet), the table understanding subsystem 114 can determine the table type of the logical sub-table, such as the type of form type or non-form type, where the non-form type can also be referred to as an ordinary table type or a standard table type, etc. Optionally, information registration forms, configuration entry forms, etc. are form types. Optionally, data reports, statistical lists, etc. are non-form types. For the area where the logical sub-table is located, the table type of the area can be identified by distinguishing the structural type. Exemplarily, the table understanding subsystem 114 can determine the table type based on features such as the distribution of empty cells, the pairing relationship of key-values, and whether there is a primary key list. It is understandable that the determination of the table type can be used for subsequent structural understanding, for example, subsequent structural understanding can be performed based on the strategy corresponding to the table type. In this way, the embodiments of the present disclosure can realize automatic recognition of table types, including distinguishing between form types and non-form types, so as to carry out targeted subsequent processing.

[0046] For example, the table understanding subsystem 114 can determine the data area and header area of ​​a logical subtable based on header detection and data area positioning. For example, the table understanding subsystem 114 can determine the data area boundaries in a logical subtable and further clean abnormal data based on characteristics such as data type and column alignment. For example, the table understanding subsystem 114 can identify the row / column area of ​​the header in the logical subtable. It is understood that the table understanding subsystem 114 in the embodiments of the present disclosure can support complex headers such as cross-row headers, multi-level headers, and slashes.

[0047] For example, the table understanding subsystem 114 can determine or mark the key columns of the logical subtable through key column identification. For example, key columns with high query value or business significance can be automatically marked based on column titles, column value distribution, and business semantics. For example, the key column can be a column with index information in the logical subtable. For example, "serial number" or "customer name" in a certain logical subtable can be determined as a key column. It is understandable that the identification of key columns can facilitate the indexing capability of subsequent processes (such as the question-answering system mentioned below).

[0048] Optionally, the grouping information may indicate which information in the worksheet (including any of the following: logical sub-tables, row description information, charts, etc.) belongs to the same group. For example, the grouping information may include information of multiple groups, wherein each group includes at least one of the following: a logical sub-table, a chart, or a description row. Specifically, the table understanding subsystem 114 may determine, based on semantics, whether the row description information and the logical sub-table correspond to the same or type of content, and then determine whether to determine the row description information and the logical sub-table as the same group. Specifically, the table understanding subsystem 114 may determine whether the chart and the logical sub-table are the same group based on the mapping relationship between the data on which the chart is based and the data in the logical sub-table. Optionally, a group may also be referred to as a semantic unit.

[0049] For example, the table understanding subsystem 114 can perform semantic-based grouping operations on various structural elements in a worksheet (such as header areas, description rows, data areas, key columns, etc.). Other parts of the semantic content belonging to a logical subtable can be aggregated with the logical subtable into a complete substructure, i.e., a group. In this way, content with the same, similar, or related semantics (such as title information, descriptive text, headers, and data areas, etc.) can be grouped into the same group. In this way, the group can be processed as an independent semantic unit in subsequent processes (such as table conversion, rendering, question-and-answer systems, etc.).

[0050] It is understandable that worksheets have a high degree of editing freedom, and a worksheet may contain content with different structures due to differences in typesetting, caused by different authors and / or input at different times. In the embodiments of the present disclosure, the grouping mechanism can effectively eliminate the differences caused by different structures, thereby improving the accuracy and interpretability of worksheet structure recognition.

[0051] In this way, the embodiments of the present disclosure can accurately identify the structural information in the table, including the division of logical sub-tables, header areas, description rows, data areas, key columns, etc., thereby improving the ability to understand tables with complex layout structures.

[0052] In some embodiments, a large language model for table understanding can be trained based on annotated data (e.g., samples). For example, after training, the large language model can have the following capabilities: end-to-end recognition of the worksheet's layout structure and output of detailed information. During training, a variety of samples can be prepared, such as tables with various complex structures. This ensures that the trained large language model can support processing a variety of table files.

[0053] For example, input information can be input into the trained large language model and output information (also called inference results) can be obtained. For example, the input information can include the position information of each cell in the worksheet and the cell content information corresponding to each position information. For example, the output information can include grouping information. Exemplarily, the output information is structured to obtain the logical sub-table area of ​​the worksheet and its detailed structure information (such as the table header, etc.). As an example, the structured output information can include:

[0054]

[0055] Based on the structured output information, it can be determined that the worksheet corresponding to the input information includes two logical sub-tables, and these two logical sub-tables are divided into two groups. The first group (the first logical sub-table) runs from row 1, column 1 to row 29, column 4, where row 1 is the header row. In addition, the first group also includes descriptive information on row 30. The second group (the second logical sub-table) runs from row 31, column 1 to row 38, column 4, where row 31 is the header row.

[0056] In this way, the structured information of the worksheet can be obtained. This structured information can be used to restore the complete structural information and semantic context information in the worksheet, providing basic data support for subsequent processes.

[0057] In this way, embodiments of the present disclosure provide a high-precision table structure understanding method, which realizes in-depth understanding of the semantic structure inside the electronic table, and can support various understanding capabilities such as identification of table types, identification and division of logical sub-tables (such as identification and binding of logical sub-tables, table header regions, data regions, description rows, key columns, etc.), semantic grouping, etc. For example, grouping based on the semantic information of each structural element of the table ensures that each logical sub-table can be involved in subsequent processing as an independent and complete semantic unit. Moreover, the present scheme can support accurate identification of various complex table structures, such as cross-row and cross-column, diagonal table headers, multi-level headers, etc.

[0058] The table format conversion subsystem 116 can store the structured information of the worksheet file in a predefined format. Specifically, for each logical sub-table of the aforementioned types that have been identified, it can be further converted into a structured and semantically clear predefined format 102. Exemplarily, the predefined format can be any one of the following: a Comma-Separated Value (CSV) format, a JavaScript Object Notation (JSON) format, a Markdown format, or a Hyper Text Markup Language (HTML) format.

[0059] Optionally, the CSV format is suitable for downstream data analysis or lightweight question-answering systems. Optionally, the JSON format can preserve the hierarchical structure and key-value semantics of the table, facilitating programmatic processing and interface transmission. Optionally, the Markdown format is suitable for human-machine collaborative review, document publishing, and other scenarios. Optionally, the HTML format is used to build web-based interactive table components, supporting direct embedding into visualization systems. In some embodiments, the predefined format can be determined / selected based on the specific scenario to be applied, so that the converted table format can meet the needs of the scenario.

[0060] Optionally, the structured information of the predefined format 102 can be referred to as a structured table, which can be stored for subsequent processes or presented to the user for easy viewing. For example, the structured table can be used in the table question-answering system 120 or in other data processing systems.

[0061] It can be understood that, although the above description is based on the example of the table question-answering system 120, the present disclosure is not limited thereto. The present disclosure can be applied to other data processing systems, such as data analysis systems, data visualization systems, etc. Figure 1The table format conversion subsystem 116 is shown as being independent of the table understanding subsystem 114. However, in other scenarios, the table understanding subsystem 114 and the table format conversion subsystem 116 may be implemented together as the same subsystem. For example, the table format conversion subsystem 116 may be an internal module of the table understanding subsystem 114, etc. This disclosure is not limited to this.

[0062] In some implementations of the present disclosure, the table processing system 110 can be integrated with the table question and answer system 120. For example, after the worksheet file 101 is structurally parsed, understood, and formatted, the structured table can be provided to the table question and answer system 120 to achieve integration.

[0063] Additionally, the form question answering system 120 may include a pre-processing module ( Figure 1 (not shown in the figure). In some embodiments, the preprocessing module can slice the structured information to determine multiple slices. Exemplarily, the preprocessing module can preprocess the structured table, for example, the preprocessing includes slicing; and further construct an index for the slice. For example, the preprocessing module can slice the logical unit, combine the header, description line, data area and other information to generate a fragment text, and generate an index by embedding the vector, and store it in the vector retrieval library. Optionally, the vector retrieval library can also be a knowledge base, a knowledge base, etc. Optionally, the vector retrieval library can be independent of the table question and answer system 120. For example, the table question and answer system 120 can access or query the vector retrieval library through an interface.

[0064] For example, if the size of the grouping information of a group is less than a threshold, the logical table and description row (if any) in the group can be used as a slice, and an index can be assigned to the slice, and the embedding vector of the slice is stored in association with the index. Optionally, a slice can also be called a fragment, a chunk, etc.

[0065] For example, if the size of the grouping information of a group exceeds a threshold, it can be divided or split into multiple slices, and an index is assigned to each slice. The embedding vector of each slice is associated with the index and stored. In addition, the size of each slice in the multiple slices does not exceed the threshold. Optionally, each slice includes header information and a description line (if any). For example, the threshold can be equal to 4096 tokens or other values.

[0066] Taking a specific group (assuming it is group 1) as an example, assuming that group 1 includes logical sub-table 1 and row description information, and logical sub-table 1 includes a header area and a data area, where the header area is 1 row and the data area is 20 rows. Slice 1 and slice 2 can be generated based on group 1, where slice 1 includes the header of logical sub-table 1, the first 10 rows of the data area of ​​logical sub-table 1, and the row description information; where slice 2 includes the header of logical sub-table 1, the last 10 rows of the data area of ​​logical sub-table 1, and the row description information.

[0067] It should be understood that the manner in which a group is sliced ​​depends on the size of the group and the semantic information of each element in the group (e.g., cells of a logical subtable, description rows, etc.). For example, slicing can be performed at a row granularity or a column granularity, which is not limited in this disclosure.

[0068] In this way, the preprocessing module can determine / generate multiple slices based on the structured table. The multiple slices and their corresponding embedding vectors can be stored in the vector search library. This facilitates operations such as searching for table information by the table question answering system 120.

[0069] The worksheet RAG module 122 is based on the RAG structure and may include a retrieval module and a generation module, and the question and answer are completed through the collaboration of the retrieval module and the generation module. Specifically, the retrieval module can perform information retrieval. According to the query information 151 input by the user, at least one slice associated with the query information 151 is retrieved from the vector retrieval library through a retrieval algorithm (such as a vector matching mechanism), etc., to form an input expected context. Specifically, the generation module can generate answers. The slice obtained by the retrieval module is input as context into the large model, so that the large model can generate accurate and targeted answer information 152 based on its own knowledge and the retrieved slices. For example, the large model can perform cross-slice semantic synthesis and question answering to obtain the final text answer, i.e., the answer information 152.

[0070] The disclosed embodiments construct a table-oriented (table-level) RAG structure. This supports rapid retrieval within a vector search library based on user query information 151 (e.g., a question), thereby identifying associated logical sub-tables and slices and generating answer content, i.e., answer information 152. This enables semantic content retrieval and answering capabilities based on structured tables.

[0071] In this way, the table question answering system 120 including the worksheet RAG module 122 can convert complex structured tables into standardized slice units and save the embedded vectors in the vector retrieval library; then the worksheet RAG module 122 can perform information extraction and multi-segment answer generation based on slices.

[0072] The worksheet agent module 124 can perform table retrieval and generate summary information. Based on the query information and the vector retrieval library, the worksheet agent module 124 determines at least one slice associated with the query information and the structured information related to the at least one slice. Specifically, the worksheet agent module 124 searches the vector database based on the query information 151 to obtain at least one slice associated with the query information 151, and can further determine the structured information related to the at least one slice. For example, the structured information related to the at least one slice can be: information including a complete group of at least one slice or information including a logical sub-table of at least one slice. The worksheet agent module 124 can generate an expected context based on the at least one associated slice retrieved from the vector retrieval library. The worksheet agent module 124 can also generate table summary information based on the structured information related to the at least one slice. For example, the table summary information can represent the summary information (or structural description information) of the group or logical sub-table corresponding to the at least one slice, so that the large model can understand the overall context.

[0073] Agents possess decision-making capabilities and can select appropriate action strategies based on different situations. The worksheet agent module 124 can determine strategies and perform task decomposition. Specifically, the large model of the worksheet agent module 124 autonomously determines strategies based on context and summary information. Strategies can also be referred to as answer strategies, and can include direct answer strategies, chain reasoning strategies, or tool invocation strategies.

[0074] For example, if the large model of the worksheet agent module 124 can determine sufficient information based on the context and summary information, a direct answer strategy can be used to generate a result, namely, answer information 152. In other words, the large model of the worksheet agent module 124 can directly generate and output answer information 152 based on the context and summary information.

[0075] For example, if the large model of the worksheet agent module 124 cannot directly determine sufficient information based on the context and summary information, then a chain reasoning strategy can be used to analyze and think through intermediate steps to further generate results (i.e., answer information 152); or a tool call strategy can be used to generate code, SQL and other tool-type instructions to assist in generating results (i.e., answer information 152).

[0076] The worksheet agent module 124 can generate and execute code to assist in generating results. Specifically, if the query information 151 may involve complex calculations and information processing, the worksheet agent module 124 can generate code and trigger an execution module to execute the code. The result of the code execution can be used as part of the answer information 152. For example, the generated code can be called query code and can include Python scripts, etc.

[0077] For example, the worksheet agent module 124 can also use a human-in-the-loop (HITL) mechanism. In some embodiments, users (e.g., system 120 developers, human experts, etc.) can modify, confirm, or enhance the code or model output generated by the large model of the worksheet agent module 124. Furthermore, the large model of the worksheet agent module 124 can further improve or update the output based on this. In this way, the worksheet agent module 124 can support scenarios where human experts intervene and further ensure the accuracy and explainability of the final answer.

[0078] In this way, the embodiments of the present disclosure can also enter the intelligent agent mechanism on the basis of RAG to achieve higher-level task planning and dynamic coping. Specifically, the table question and answer system including the worksheet intelligent agent module 124 can further enhance the task execution capability, including table summary generation, policy decision-making (direct answer, chain reasoning, code generation), tool calling and execution, etc., further improving the interactive ability and utilization efficiency of table data. In addition, by introducing the HILP mechanism, the reliability and controllability of the answer can be further improved. For example, the large model of the worksheet intelligent agent module 124 can perform reasoning planning between table summaries, historical interactions and user questions, dynamically determine the optimal problem-solving path, integrate retrieval, calling, calculation and reasoning capabilities, and improve the intelligence level of the system.

[0079] In some embodiments of the present disclosure, the form question answering system 120 may further include a selection module (not shown), for example, the selection module is configured to select the worksheet RAG module 122 or the worksheet agent module 124 .

[0080] For example, the user input may include query information 151 and indication information, wherein the indication information indicates RAG (or worksheet RAG module 122) or agent (worksheet agent module 124). Accordingly, the selection module may select the corresponding module based on the indication information.

[0081] It should be noted that the embodiments of the present disclosure do not limit the specific content of the table. Figure 1The illustrated system 100 can be deployed on an enterprise's server. Accordingly, the content of the worksheet files can depend on the enterprise. For example, if the enterprise is a school, the worksheet files may include, but are not limited to, class schedules, student personal information registration forms, class information registration forms, teacher information registration forms, semester transcripts, and financial usage registration forms. Alternatively, the worksheet files may also include, but are not limited to, purchase / usage registration forms for various stationery, books, and equipment.

[0082] Figure 2 A schematic flow chart of a process 200 for question answering according to some embodiments of the present disclosure is shown. Figure 2 The process 200 can be implemented by the aforementioned form question answering system 120. For example, it can be implemented by an electronic device or a cloud server.

[0083] At 210 , input information from a user may be received, the input information including query information and instruction information.

[0084] Optionally, the query information may indicate the content to be queried or retrieved, or a question posed by the user. For example, the query information may be: calculate the average of the grade column in Table A of the transcript. For example, the query information may be: provide a comparison of the financial statements for 2023 and 2024.

[0085] Optionally, the indication information may indicate a RAG or an agent. For example, the indication information being "1" indicates a RAG. For example, the indication information being "0" indicates an agent.

[0086] Exemplarily, if the indication information indicates a RAG, the process further executes 220 ; if the indication information indicates an agent, the process further executes 230 .

[0087] At 220 , at least one slice associated with the query information is determined from the vector search library.

[0088] At 240, answer information is generated based on the at least one slice using the large model of the worksheet RAG module. In this branch, the model output may include the answer information.

[0089] In another branch, at 230 , at least one slice associated with the query information is determined from the vector retrieval library, and structured information is determined. Furthermore, table summary information of the table corresponding to the structured information can also be automatically generated.

[0090] At 250, code is generated from a macro model of a worksheet agent module based on at least one slice and the table summary.

[0091] At 270, code is executed to assist in generating answer information. In this embodiment, the model output may include answer information and code.

[0092] Understandably, Figure 2 The process 200 is merely illustrative and is not intended to limit the scope of protection of the present disclosure. In some examples, if the strategy determined by the worksheet agent module is to answer directly or through an intermediate step, then the above-mentioned step of generating code can be omitted, and the model output accordingly does not include code.

[0093] For example, the selection module may also select the worksheet RAG module 122 or the worksheet agent module 124 based on the semantics or type of the query information 151. For example, if the query information 151 is of a retrieval type, the worksheet RAG module 122 may be selected. For example, if the query information 151 is of an analysis type, the worksheet agent module 124 may be selected. Alternatively, for query information 151 whose type cannot be determined, the worksheet agent module 124 may be selected.

[0094] The technical solutions provided by the embodiments of the present disclosure significantly improve the structural analysis and semantic understanding capabilities of complex tabular data, and provide effective support for applications such as intelligent question answering and decision support based on tabular data.

[0095] Figure 3 FIG3 is a schematic flow chart of a table processing method 300 according to some embodiments of the present disclosure. The method 300 may be executed by an electronic device or a cloud server, or may be executed by a processor.

[0096] At block 310 , a worksheet file to be processed is obtained. At block 320 , structural information of the worksheet file is determined using a large language model for table understanding. The structural information includes at least one of the following for a worksheet in the worksheet file: information about logical subtables, row descriptions, or grouping information. At block 330 , the structural information is stored using a predefined format.

[0097] In some embodiments, the worksheet file to be processed may be a file including a spreadsheet stored in a storage system of an enterprise. Exemplarily, the source of the worksheet file may be any application. Exemplarily, the worksheet file may include one or more worksheets.

[0098] In some embodiments, determining the structured information of the worksheet file using the large language model for table understanding includes: parsing the worksheet file to extract information about the worksheet in the worksheet file; and determining the structured information based on the worksheet information using the large language model.

[0099] For example, referring to the above combination Figure 1The table parsing subsystem 112 can parse the worksheet file to extract the worksheet information in the worksheet file. The table understanding subsystem 114 can use a large language model to determine the structured information based on the worksheet information. For example, the large language model is a large language model trained for table understanding. In addition, the table format conversion subsystem 116 can use a predefined format to store the structured information. For example, the predefined format can be any of the following: CSV format, JSON format, Markdown format, or HTML format.

[0100] Exemplarily, the information of a worksheet includes at least one of the following: whether row data is visible or hidden, whether column data is visible or hidden, filtering information for column data, text, numbers, or formulas in cells, cell attribute information, expressions and calculation results of formulas in cells, cell comments and / or notes, cell merge information and logical layout, or embedded charts and mapping relationships between charts and data. Exemplarily, the information of a logical subtable includes at least one of the following: the table type of the logical subtable in the worksheet, the area where the logical subtable is located, the data area of ​​the logical subtable, the header area of ​​the logical subtable, or the key columns of the logical subtable. Exemplarily, the row description information includes information describing rows outside the logical subtable. Exemplarily, the grouping information includes information of multiple groups, wherein each group includes at least one of the following: a logical subtable, a chart, or a logical row.

[0101] For detailed information about the worksheet, please refer to the Figure 1 For the sake of brevity, some descriptions are not repeated here.

[0102] In some embodiments, the stored structured information of the worksheet file, for example, structured information stored in a predetermined format, can be used for subsequent processing, for example, can be used in a form question answering system.

[0103] In some embodiments, the structured information may be sliced ​​to determine multiple slices, where the size of each slice does not exceed a threshold size; and the embedding vectors of the multiple slices are stored in a vector search library. Exemplarily, the vector search library also stores indexes of the corresponding slices. For example, an index may be assigned to each slice (or each embedding vector).

[0104] In some implementations, a query message may be received from a user; at least one slice associated with the query message may be retrieved from a vector search library; a corpus context may be generated based on the at least one slice; and answer information corresponding to the query message may be generated using a large model based on the corpus context. For example, the worksheet RAG module 122 may generate the answer information corresponding to the query message.

[0105] In some implementations, a query message may be received from a user; at least one slice associated with the query message may be retrieved from a vector search library; structured information related to the at least one slice may be determined based on the query message and the vector search library; table summary information may be generated based on the structured information related to the at least one slice; and answer information corresponding to the query message may be generated using a large model based on the corpus context and the table summary information. For example, the worksheet agent module 124 may generate the answer information corresponding to the query message.

[0106] Exemplarily, the query code may be generated based on the query information and using the large model; wherein the answer information is generated based on the query code, and wherein the output of the large model includes the answer information and the query code. Exemplarily, the query code may be generated by the worksheet agent module 124.

[0107] Exemplarily, the method may further include: obtaining input information from the user regarding the query code, wherein the input information is modification, confirmation, or enhancement; and updating the large model based on the input information. In this way, the large model can be updated through the HITL mechanism.

[0108] Understandably, about Figure 3 The specific implementation of the method 300 can refer to the above combination Figure 1 The embodiments described in detail with respect to the system 100 are not repeated here for the sake of brevity.

[0109] Through the embodiments of the present disclosure, a complete table processing and semantic understanding solution is proposed, which improves the availability of table data in the question-answering system. Specifically, the processing and semantic understanding of tables include but are not limited to: identification of table types, identification and splitting of logical sub-tables, identification and extraction of header areas and data areas, determination of grouping, etc. Based on this, a table with a complex structure can be parsed into multiple groups with clear semantics and regular structures. Furthermore, slicing processing, vectorized storage, etc. can also be performed, thereby providing a high-quality retrieval basis for the question-answering system, thereby significantly improving the availability and accuracy in real business.

[0110] It should be understood that in the embodiments of the present disclosure, "first", "second", "third", etc. are only used to indicate that multiple objects may be different, but at the same time do not exclude that two objects are the same, and should not be interpreted as any limitation on the embodiments of the present disclosure.

[0111] It should also be understood that the division of the modes, situations, categories and embodiments in the embodiments of the present disclosure is only for the convenience of description and should not constitute a special limitation. The features in various modes, categories, situations and embodiments can be combined with each other when it is logical.

[0112] It should also be understood that the above content is only intended to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Those skilled in the art may make various modifications, variations, or combinations based on the above content. Such modifications, variations, or combinations are also within the scope of the embodiments of the present disclosure.

[0113] It should also be understood that the description of the above content focuses on emphasizing the differences between the various embodiments, and the same or similar points can be referenced or borrowed from each other. For the sake of brevity, they will not be repeated here.

[0114] Figure 4 1 shows a schematic block diagram of an example apparatus 400 according to some embodiments of the present disclosure. The apparatus 400 may be implemented in software, hardware, or a combination of both. Figure 4 As shown, the apparatus 400 includes an acquiring unit 410 , a determining unit 420 and a storing unit 430 .

[0115] The acquisition unit 410 is configured to acquire a worksheet file to be processed. The determination unit 420 is configured to determine structured information of the worksheet file using a large language model for table understanding. The structured information includes at least one of the following for a worksheet in the worksheet file: information about a logical subtable, row description information, or grouping information. The storage unit 430 is configured to store the structured information using a predefined format.

[0116] In some embodiments, apparatus 400 further includes a processing unit configured to slice the structured information to determine a plurality of slices, wherein the size of each slice does not exceed a threshold size. Apparatus 400 may also include a second storage unit configured to store the embedding vectors of the plurality of slices in a vector search library. Optionally, the vector search library also stores indexes of the corresponding slices.

[0117] In some embodiments, the device 400 may include a receiving unit configured to receive query information from a user; a retrieval unit configured to retrieve at least one slice associated with the query information from a vector retrieval library; a generation unit configured to generate a corpus context based on the at least one slice; and based on the corpus context, using a large model, generate answer information corresponding to the query information.

[0118] Exemplarily, the generation unit is further configured to determine structured information related to at least one slice based on the query information and the vector search library; and generate tabular summary information based on the structured information related to the at least one slice. Optionally, the generation unit is configured to generate answer information corresponding to the query information using a large model based on the corpus context and the tabular summary information.

[0119] Exemplarily, the generating unit is further configured to generate a query code based on the query information and using the big model; wherein the answer information is generated based on the query code, and wherein the output of the big model includes the answer information and the query code.

[0120] In some embodiments, the receiving unit may be further configured to: obtain input information of the query code from the user, where the input information is modification, confirmation, or enhancement. The processing unit may be further configured to: update the macro model based on the input information.

[0121] In some embodiments, the determining unit is configured to: parse the worksheet file to extract information about the worksheet in the worksheet file; and determine the structured information based on the worksheet information using a large language model.

[0122] Exemplarily, the information of a worksheet includes at least one of the following: whether row data is visible or hidden, whether column data is visible or hidden, filtering information of column data, text, numbers, or formulas in cells, cell attribute information, expressions and calculation results of formulas in cells, cell comments and / or remarks content, cell merging information and logical layout, or embedded charts and the mapping relationship between charts and data.

[0123] Illustratively, the information of the logical subtable includes at least one of the following: the table type of the logical subtable in the worksheet, the area where the logical subtable is located, the data area of ​​the logical subtable, the header area of ​​the logical subtable, or the key column of the logical subtable.

[0124] Exemplarily, the row description information includes information describing the row outside the logical sub-table.

[0125] Exemplarily, the grouping information includes information of a plurality of groups, wherein each group includes at least one of the following: a logical sub-table, a chart, or a description line.

[0126] Figure 4 The device 400 can be used to achieve the above combination Figures 1 to 3 For the sake of brevity, the above process will not be described in detail here.

[0127] The division of modules or units in the embodiments of the present disclosure is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the disclosed embodiments may be integrated into a single unit, exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0128] Figure 51 shows a block diagram of an example computing device 500 that can be used to implement embodiments of the present disclosure. It should be understood that Figure 5 The computing device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figures 1 to 3 The process described.

[0129] like Figure 5 As shown, computing device 500 is in the form of a general-purpose computing device. Components of computing device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 500.

[0130] The computing device 500 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 500.

[0131] The computing device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0132] The communication unit 540 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 500 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0133] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Computing device 500 may also communicate with one or more external devices (not shown) via communication unit 540, as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with computing device 500, or with any device that allows computing device 500 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0134] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0135] The computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0136] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0137] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all.

[0138] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0139] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A form processing method, comprising: Get the worksheet file to be processed; Determining structural information of the worksheet file using a large language model for table understanding, wherein the structural information includes at least one of the following for a worksheet of the worksheet file: information of a logical subtable, row description information, or grouping information; and The structured information is stored using a predefined format.

2. The method according to claim 1, further comprising: Slicing the structured information to determine a plurality of slices, wherein a size of each slice does not exceed a threshold size; as well as The embedding vectors of the plurality of slices are stored in a vector retrieval library.

3. The method according to claim 2, wherein the vector search library also stores an index of the corresponding slice.

4. The method according to claim 2, further comprising: Receive query information from users; Retrieving at least one slice associated with the query information from the vector retrieval library; generating a corpus context based on the at least one slice; as well as Based on the corpus context, answer information corresponding to the query information is generated using a large model.

5. The method according to claim 4, further comprising: Determining structured information related to the at least one slice based on the query information and the vector search library; as well as Tabular summary information is generated based on the structured information related to the at least one slice.

6. The method according to claim 5, wherein generating answer information corresponding to the query information based on the corpus context and using a large model comprises: Based on the corpus context and the table summary information, the answer information corresponding to the query information is generated using the large model.

7. The method according to claim 6, further comprising: Based on the query information; Generate query code using the large model; The answer information is generated based on the query code, and the output of the large model includes the answer information and the query code.

8. The method according to claim 7, further comprising: Obtaining input information of the query code from the user, wherein the input information is modification, confirmation, or enhancement; as well as Based on the input information, the large model is updated.

9. The method according to claim 1, wherein determining the structured information of the worksheet file by using a large language model for table understanding comprises: Parsing the worksheet file to extract worksheet information from the worksheet file; as well as The structured information is determined based on the information of the worksheet using the large language model.

10. The method according to claim 9, wherein the information of the worksheet includes at least one of the following: Row data is visible or hidden, Column data is visible or hidden, Filter information for column data, The text, numbers, or formulas in the cells, Cell attribute information, The expression and calculation result of the formula in the cell, Cell comments and / or notes content, Cell merging information and logical layout, or Embedded charts and the mapping relationship between charts and data.

11. The method according to claim 1, wherein the information of the logical sub-table includes at least one of the following: the table type of the logical sub-table in the worksheet, the area where the logical sub-table is located, the data area of ​​the logical sub-table, the header area of ​​the logical sub-table, or the key column of the logical sub-table.

12. The method according to claim 1, wherein the row description information includes information describing the row outside the logical sub-table.

13. The method of claim 1, wherein the grouping information comprises information of a plurality of groups, wherein each group comprises at least one of the following: a logical sub-table, a table, or a description line.

14. An electronic device comprising: at least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to any one of claims 1 to 13.

15. A form processing device, comprising: an acquiring unit, configured to acquire a worksheet file to be processed; a determining unit configured to determine structural information of the worksheet file using a large language model for table understanding, wherein the structural information includes at least one of the following for a worksheet of the worksheet file: information of a logical subtable, row description information, or grouping information; as well as The storage unit is configured to store the structured information using a predefined format. 16 . A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to claim 1 is implemented.

Images