Rag-based table data processing method and system

Abstract

The application provides a RAG-based table data processing method and system, the method comprises the following steps: obtaining a table analysis path in a file; processing content data in the table; receiving the processed content data, and establishing a mixed index suitable for different table structures of sparse tables and multi-level tables, and a context joint index of the content data in the table and a merged index of adjacent files; obtaining user query content, converting the user query content into a key-value combination, and performing vector retrieval and keyword retrieval; according to the results of the key-value combination, the vector retrieval and the keyword retrieval, performing dynamic template filling, guiding the generation of instructions by a data fine-tuning model, and simultaneously generating lightweight calculation results; combining query delay and accuracy, and adjusting the mixed index and the context joint index into rows, columns or sub-tables. The system comprises an analysis conversion module, a retrieval adaptation module and a generation integration module. The application solves the contradiction between query flexibility and result accuracy in traditional table processing methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of structured data processing, and in particular to a table data processing method and system based on RAG (Retrieval-Augmented Generation). BACKGROUND

[0002] In 2020, Meta (formerly Facebook AI) first proposed a retrieval-augmented generation framework, combining non-parametric retrieval (such as BM25 or Dense Retrieval) with parametric generation models (such as BART). Early related models include Meta's retrieval-augmented generation (BM25 or Dense Retrieval + BART generation), Google's pre-training and fine-tuning of knowledge-intensive tasks, and later advanced models include independent encoding of retrieval documents, improving generation efficiency, model autonomous control of retrieval timing and content, and enhancing controllability. The overall evolution direction is as follows: (1) Model architecture: from single retrieval to generation pipeline to end-to-end joint optimization (such as RAG-Token and RAG-Sequence); (2) Retrieval technology: from sparse retrieval (BM25) to dense retrieval (such as DPR, ANCE); (3) Application expansion: from text question answering to multi-modal, dialogue system, etc. The existing technical solution mainly identifies the rows and columns of the table, structures the table rows and columns, describes them as "key-value pairs", then converts the key-value pairs into describable text, and finally indexes the table cells to facilitate data retrieval and generation. The main shortcomings are as follows: The complex situations such as merged cells and multi-level table headers are not considered, and the table parsing result may not be complete. The context of the table is not considered, and the parsing of the table may be fragmented. Only the table cells are indexed, which is difficult to support complex queries. The model retrieval method is relatively single, and it is difficult to quickly retrieve the required results. The support capability for numerical or statistical data queries is weak, and complex query calculations cannot be supported. There is a lack of necessary processing for noise in the table, such as irrelevant annotations, empty cells, or format errors. There is a lack of evaluation and optimization mechanism, and the retrieval accuracy and generation result correctness are difficult to continuously improve.

[0003] Prior art one, application number: CN202411046896.7 discloses a table data processing method and device, the method comprises: listening to the scroll action of the scroll bar of the current page of the table list and recording the real-time scroll distance value LR of the scroll bar, and acquiring a first control vector K1 in real time;Set the second control vector K2 corresponding to the table data;Iterate through all elements contained in the table data, for each element EMi, update the cumulative height AHi=AHi-1+Hi;Form a loading ternary number SJ={TJ,VG,KP}, determine the loading strategy of CLA data, PT value and PB value based on the ternary number SJ. Although the matching smooth display data is set according to the different operation modes of the current data, so as to avoid operation lag;But the index mode is single, only according to the scroll speed and cumulative height to adjust the data loading strategy, without considering the structure sparsity or multi-level nested relationship of the table, the efficiency is insufficient when querying complex data;Weak data processing capacity, without enhancing, semantic or standardizing the data, it is difficult to deal with unstructured or cross-file table data association query.

[0004] Prior art two, application number: CN202410793445.3 discloses a table data processing method, device, equipment and storage medium, the method comprises: acquiring an original file;According to the original file, generate file data and node data corresponding to the original file;According to the file data, determine the first data corresponding to the original file;According to the node data, determine the second data corresponding to the original file;According to the first data and the second data, create an online table, get an online editable table file. Although it can efficiently and accurately manage table data, meet the needs of modern laboratory for table data processing and generation, help table data collaboration and long-term preservation;But lack of intelligent retrieval ability, only rely on the storage management of the original file node data, without introducing vector retrieval or dynamic index optimization mechanism, the support for complex query is weak;Cross-file association retrieval is difficult, the scheme does not involve hybrid index construction, and cannot optimize the association query efficiency of multiple files through context joint index.

[0005] The prior art three, application number: CN202410797889.4 discloses a table data processing method, device, equipment and storage medium, in response to the triggering operation of the field control in the table page, the field interface is displayed; in the field interface, the field component and the first field are determined; according to the field component and the first field, the second field is displayed in the table page, and the second data is generated in the data column corresponding to the second field, the second data is based on the first data in the data column corresponding to the first field, and the data generated by the field component. Although the field type of the second field is determined based on the field component, the use efficiency of the user is improved; however, the automation processing capacity is limited, although the data can be dynamically generated according to the user operation, the data enhancement or structured processing method is not included, and the complex table of different formats cannot be adapted; the query mechanism is missing, the index optimization strategy is not established for the table data, and the field is generated only by manual operation, and efficient multidimensional retrieval is not supported.

[0006] At present, the prior art one, the prior art two and the prior art three have the problems of low query efficiency, poor structure adaptability and difficult cross-file association in complex table data processing. Therefore, the present application provides a table data processing method and system based on RAG. SUMMARY

[0007] In order to solve the above technical problems, the present application provides a table data processing method based on RAG, which comprises the following steps:

[0008] The user query content is converted into a key-value combination, vector retrieval and keyword retrieval are performed; according to the results of key-value combination, vector retrieval and keyword retrieval, dynamic template filling is performed, and the instruction data fine tuning model is guided to generate instructions.

[0009] Optionally, the content data in the table is subjected to enhancement processing, structured processing, semantic processing and standardization processing to obtain processed content data.

[0010] Optionally, the process of obtaining the processed content data comprises the following steps:

[0011] A storage library for storing the parsing program is created;

[0012] When a file with structural characteristics is obtained, an instruction for triggering the parsing program is issued, and the electronic file format and the parsing mode are confirmed in sequence according to the identification process of the parsing program.

[0013] Optionally, when the electronic file format is confirmed, the parsing is performed according to the corresponding interpretation mode; the electronic file format is taken as a starting node, and the parsing mode is taken as a terminal node, thereby forming a table parsing path in the file.

[0014] Optionally, the electronic file format is a file extension, and the file includes word, excel and pdf.

[0015] Optionally, after forming a table parsing path in the file, metadata, table headers, data in each row and numerical date content data in the table are obtained, and different content data is subjected to enhancement processing, structured processing, semantic processing and standardized processing.

[0016] Optionally, the parsing program includes an electronic file format and a corresponding parsing method, and a file with structural characteristics that triggers the parsing program.

[0017] Optionally, according to the test case, the query delay and the accuracy are combined to adjust the mixed index and the context joint index to the row, the column or the sub-table.

[0018] Optionally, the process of adjusting the mixed index and the context joint index to the row, the column or the sub-table includes the following steps:

[0019] Based on the joint feedback result of vector retrieval and keyword retrieval, the distribution of blank areas of the sparse table is quantitatively analyzed; according to the processed table header hierarchical relationship, the topological features of the sub-table are extracted; when the user query key value combination involves multi-layer association, the original single-level index is automatically disassembled into a tree-like sub-table index network, the generated lightweight calculation result is used as a verification data set, and through comparison of the delay-accuracy curve of the test case, three-stage optimization is implemented.

[0020] The application provides a table data processing system based on RAG, which comprises:

[0021] The analysis and conversion module is responsible for enhancing, structuring, semantic processing and standardizing the content data in the table to obtain processed content data.

[0022] The retrieval adaptation module is responsible for receiving the processed content data, establishing a mixed index and a context joint index, obtaining user query content, converting the key value combination, and performing vector retrieval and keyword retrieval.

[0023] The generation integration module is responsible for dynamically filling a template according to the results of the key value combination, vector retrieval and keyword retrieval, and guiding the generation of instructions by adjusting the data model.

[0024] The present application realizes an end-to-end table intelligent processing pipeline through the deep coupling of multi-stage technical features, establishes a unified feature representation of cross-modal table content through file format adaptive analysis path selection and multiple processing (metadata enhancement, flattening / nested structure, semantic conversion and standardization) of content data, combines a hybrid index architecture (fine-grained cell index and graph relationship storage) and a context joint index mechanism to construct a unified index space with multi-dimensional retrieval capability, realizes accurate mapping of query intent to structured results based on the collaborative optimization of dynamic query conversion (key-value combination / vector mapping / keyword matching) and generation components (template engine / lightweight calculation / fine-tuning model), and finally forms a closed-loop processing system with adaptive capability by continuously optimizing the index granularity relying on the feedback regulation mechanism of delay-accuracy, thereby overcoming the key technical bottlenecks such as semantic discontinuity, low retrieval efficiency and insufficient generation availability of traditional methods in processing multi-source heterogeneous table data.

[0025] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the structure particularly pointed out in the written description and the accompanying drawings.

[0026] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, which together with the embodiments of the present application serve to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0028] Figure 1 A flow chart of the table data processing method based on RAG in embodiment 1 of the present application;

[0029] Figure 2 A principle diagram of the table data processing method based on RAG in embodiment 1 of the present application;

[0030] Figure 3 A process diagram of the processed content data in embodiment 2 of the present application;

[0031] Figure 4 A table data processing process diagram of hybrid index and joint retrieval in embodiment 5 of the present application;

[0032] Figure 5 A process diagram of adjusting the hybrid index and the context joint index to rows, columns or sub-tables in embodiment 9 of the present application;

[0033] Figure 6A RAG-based table data processing system block diagram in embodiment 10 of the present application. DETAILED DESCRIPTION

[0034] The preferred embodiments of the present application will be described herein below with reference to the accompanying drawings, in which it is understood that the preferred embodiments described herein are merely for the purpose of illustrating and explaining the present application and are not intended to limit the present application.

[0035] The terminology used in the present application is merely for the purpose of describing particular embodiments and is not intended to limit the present application. As used in the present application, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refer to and encompass any or all possible combinations of one or more of the associated listed items.

[0036] The following description refers to the accompanying drawings. Wherever possible, the same reference numbers in different drawings refer to the same or similar elements. The following exemplary embodiments are described hereinbelow with reference to the drawings. The embodiments described herein are merely exemplary of the present application and are not intended to limit the present application. Rather, they are intended to explain and describe the devices and methods consistent with the present application. In the description of the present application, it should be understood that the terms "first," "second," "third," etc., are used merely to distinguish similar objects from one another, and are not necessarily used to describe a particular sequential or chronological order, nor are they used to indicate or imply relative importance of the objects. The specific meanings of the above terms in the present application can be understood by those skilled in the art according to the specific circumstances.

[0037] Embodiment 1: As shown in the present application, a RAG-based table data processing method is provided, comprising the following steps: Figure 1

[0038] S100: Obtain a file with structured characteristics and its electronic file format, and obtain the table parsing path in the file from the pre-stored parsing program according to the electronic file format; perform enhancement processing, structured processing, semantic processing and standardization processing on the content data in the table to obtain processed content data; wherein the content data includes metadata, table headers, data in each row and digital dates of the table;

[0039] S200: Receive the processed content data, establish a mixed index suitable for different table structures of sparse tables and multi-level tables, and a context joint index of the content data in the table and the adjacent file merge index; obtain user query content, convert it into a key-value combination, and perform vector retrieval and keyword retrieval;

[0040] ​S300: According to the results of key-value combination, vector retrieval and keyword retrieval, dynamic template filling is performed, instruction data fine-tuning model is guided to generate instructions, and lightweight calculation results are generated; according to the test case, the query delay and accuracy are combined to adjust the mixed index and the context joint index to the row, column or sub-table.

[0041] The working principle and beneficial effects of the above technical solution are: first, the structured characteristic file and its electronic file format (file extension) are obtained, the table parsing path in the file is obtained from the pre-stored parsing program according to the electronic file format; the content data in the table is enhanced, structured, semantized and standardized to obtain the processed content data; the content data includes metadata, table header, each row of data and digital date; second, the processed content data is received, the mixed index suitable for different table structures of sparse table and multi-level table and the context joint index of the content data in the table and the adjacent file merge index are established; the user query content is converted into key-value combination for vector retrieval and keyword retrieval; finally, according to the results of key-value combination, vector retrieval and keyword retrieval, dynamic template filling is performed, instruction data fine-tuning model is guided to generate instructions, and lightweight calculation results are generated; according to the test case, the query delay and accuracy are combined to adjust the mixed index and the context joint index to the row, column or sub-table (for specific principles, refer to the attached Figure 2 ) document). The above scheme realizes an end-to-end table intelligent processing pipeline through the deep coupling of multi-stage technical features, establishes a unified feature representation of cross-modal table content through adaptive file format parsing path selection and multiple content data processing (metadata enhancement, flat / nested structure, semantic conversion and standardization), constructs a unified index space with multi-dimensional retrieval capability through the mixed index architecture (fine-grained cell index and graph relationship storage) and the context joint index mechanism, realizes the accurate mapping of query intent to structured results based on the collaborative optimization of dynamic query conversion (key-value combination / vector mapping / keyword matching) and generation components (template engine / lightweight calculation / fine-tuning model), and continuously optimizes the index granularity relying on the delay-accuracy feedback regulation mechanism to finally form a closed-loop processing system with adaptive capability, which overcomes the key technical bottlenecks such as semantic discontinuity, low retrieval efficiency and insufficient generation availability in processing multi-source heterogeneous table data in traditional methods.

[0042] The application mainly designs a data processing strategy and method for table data with structured characteristics in word, excel and pdf files in the RAG (Retrieval Augmented Generation) engineering. The core idea is to retrieve relevant information from the knowledge base to assist in generating models to produce more accurate and factual text output. It can be applied to open domain question answering, dialogue systems, content generation and other scenarios, and can convert table data from "static storage" to "dynamic knowledge assets". At the same time, it has universality and can be flexibly extended to the field of large language models (LLM: Large Language Model).

[0043] Embodiment 2: as shown in Figure 3 Based on embodiment 1, the process for obtaining processed content data provided by the application embodiment includes the following steps:

[0044] S101: create a repository for storing parsing programs, which contain electronic file formats and corresponding parsing methods, and files with structural characteristics that trigger the parsing program;

[0045] S102: when the file with structural characteristics is obtained, the instruction for triggering the parsing program is issued, and the confirmation of the electronic file format and the parsing method is carried out in turn according to the identification process of the parsing program; when the electronic file format is confirmed, the parsing is carried out according to the corresponding interpretation method; taking the electronic file format as the starting node and the parsing method as the terminal node, the table parsing path in the file is formed;

[0046] S103: obtain the metadata, table header, each row of data and digital date in the table, and carry out enhancement processing, structured processing, semantic processing and standardization processing according to different content data.

[0047] The working principle and beneficial effects of the technical solution are as follows: the embodiment first creates a storage library for storing a parsing program, the parsing program including an electronic file format and a corresponding parsing method, and a file with structural characteristics triggering the parsing program; secondly, when the file with structural characteristics is obtained, an instruction triggering the parsing program is issued, and the electronic file format and the parsing method are confirmed in sequence according to the identification process of the parsing program; when the electronic file format is confirmed, the corresponding interpretation method is used for parsing; the electronic file format is used as a starting node, and the parsing method is used as a terminal node, to form a table parsing path in the file; finally, the content data such as metadata, table headers, each row of data and digital dates in the table are obtained, and different content data are subjected to enhancement processing, structured processing, semantic processing and standardized processing. The above scheme uses the storage library to preset and dynamically trigger a unified parsing entrance independent of the format, uses the file structure characteristics as a parsing routing key, realizes end-to-end automatic extraction from the original file to the content entity (table / metadata, etc.), and ensures that the format identification (starting node) and the parsing method (terminal node) form a deterministic mapping relationship through the graph structure modeling of the parsing path, so that the redundant calculation of the traditional conditional branch parsing is eliminated. The intermediate representation generated based on the table parsing path drives the content extraction engine, realizes the decoupling of the physical storage format to the logical data unit (row / column / field) layer by layer, integrates the four-stage processing of enhancement / structuring / semantic / standardization to form a data quality improvement pipeline, so that the original unstructured data reaches the standardized state of machine readability. Through the explicit construction of the parsing path and the multi-dimensional processing of the content data, a complete audit chain including the original format characteristics, the parsing decision log and the data conversion history is formed; the intermediate results are transmitted between steps through a unified data bus, so that the data consistency and the backtracking ability of the processing process are ensured.

[0048] In the embodiment, the table parsing and the processing process of the content data thereof include the following steps:

[0049] File parsing (parsing method for different types of files):

[0050] Table parsing in a PDF file (using pdfplumber), traversing each page of the PDF file, detecting the table area in the page (based on visual features such as lines, spacing and text alignment), extracting cell text content, and generating a logical table structure (table.cells) according to the table structure (row and column spacing, border);

[0051] Merged cell processing: identifying physically merged cells (horizontally or vertically merged), which can be automatically restored by page.extract_table(merge_cells=True);

[0052] If automatic fault identification (such as no border merging unit), you can use page.rects to calculate the overlapping area, manually reconstruct the table skeleton;

[0053] Multi-level table header processing flow (such as business report): Detect table header level (such as upper title spanning columns, lower table header subdivision); Reconstruct flat structure (such as converting "year→quarter" to "2023_Q1").

[0054] Excel / CSV parsing (using pandas):

[0055] Structured table reading: Excel files use pd.read_excel(), CSV files use pd.read_csv(), support custom separators (such as \t or ;) and encoding detection (such as UTF-8 / GBK);

[0056] Merged cell processing: Parse Excel merged areas (merged_cells attribute), automatically fill repeated values (ffill()); Multi-level table header processing: Identify multi-level column index (MultiIndex), reorganize into single-layer column names with hierarchical separators (such as "sales.2023.Q1").

[0057] Metadata enhancement (table context extraction):

[0058] Extract structured information to enhance table retrieval and understanding, including: basic metadata, file name, page number, table chapter (PDF bookmark or chapter title);

[0059] Table-specific metadata: title detection, text above adjacent tables (NLP matching such as "Table 1:" or regular / Table[\s\d]+ / i);

[0060] Source and time, footer, document properties (PDF doc.info), or adjacent text (such as "Data source: National Bureau of Statistics").

[0061] Structured processing (table data conversion):

[0062] Simple table (single-level table header): Convert to key-value pair form (entity, attribute, value); Applicable scenarios: database storage (such as RDF), search engine inverted index (Elasticsearch);

[0063] Complex table (multi-level table header nesting), recursively disassemble levels to generate nested JSON structure; Applicable scenarios: time series data analysis, BI tools (such as Power BI hierarchical drilling).

[0064] Semantic processing (natural language conversion):

[0065] Convert table content to readable text, enhance semantic understanding:

[0066] Single-row description generation (for sparse tables): Template output "Product A's specifications are 25ml, with sales of 100 bottles." Multi-level table summary (for complex reports): Generate summary in context, "In the first quarter of 2023, the production of product A was 100 bottles; in the same period of 2024, it increased to 120 bottles."

[0067] Context enhancement technology:

[0068] Inject domain knowledge; Large Language Model (LLM) fine-tuning generates coherent descriptions.

[0069] Data standardization (format unification and cleaning):

[0070] Date standardization: Parse mixed formats (such as "2023 / 01 / 01", "Jan-1-2024") → unified to ISO format YYYY-MM-DD;

[0071] Number formatting: unify thousandth (1,000→1000), scientific notation (1.2E3→1200); unit normalization (such as "25ml"→"25 milliliters");

[0072] Null and noise processing: remove meaningless placeholders (such as "-", "N / A");

[0073] Filling strategy: adjacent row mean (numeric), high-frequency item (categorical data);

[0074] Output structured paradigm: adapt to target systems (such as SQL tables, JSON-LD).

[0075] Example 3: Based on example 2, the parsing program provided by the embodiment of the present application further comprises a processing process for merging cells and multi-level table headers, comprising the following steps:

[0076] S1021: The table merges multiple adjacent cells into a large cell, identifies the structure boundary of the table by analyzing lines or blank areas, judges whether there is a missing physical division line, checks the continuity of cell content, if the text of a certain cell spans multiple rows and columns, but the adjacent area lacks clear separation markers, it may imply a merging relationship; Based on the position relationship of the cells, a topological network is established to detect whether there is continuous blank or unreasonable row and column distribution to infer potential merging areas; combine boundary detection and semantic analysis results to automatically merge related cells;

[0077] S1022: The data attributes of the multi-layer table header are distributed in a tree shape, the hierarchical relationship of the table header is judged based on the difference of the font and the offset of the relative space, each level of the table header is scanned through a recursive mode, the nested header attributes are converted into single flat levels, and the original hierarchical relationship is preserved in the compression process, and a specific coding or logical index mechanism is used;

[0078] S1023: The resolution strategy is dynamically adjusted according to the visual flow direction of the table content, when a cell spans multiple logical areas, the content distribution and the space weight are comprehensively evaluated, and a division scheme is selected, and the stress distribution of the internal data when the cell is split or merged is calculated, and a structure correlation strategy is formulated.

[0079] The working principle and beneficial effects of the above technical solution are as follows: firstly, the table merges multiple adjacent cells into a large cell, analyzes the lines or blank areas, identifies the structural boundaries of the table, judges whether there is a missing physical division line, checks the continuity of the cell content, if the text of a certain cell spans multiple rows and multiple columns, but the adjacent area lacks clear separation marks, it may imply a merging relationship, a topological network is established based on the position relationship of the cells, whether there is continuous blank or unreasonable row and column distribution is detected, and the potential merging area is inferred, the related cells are automatically merged combining the boundary detection and semantic analysis results; secondly, the data attributes of the multi-layer table header are distributed in a tree shape, the hierarchical relationship of the table header is judged based on the difference of the font and the offset of the relative space, each level of the table header is scanned through a recursive mode, the nested header attributes are converted into single flat levels, and the original hierarchical relationship is preserved in the compression process, and a specific coding or logical index mechanism is used; finally, the resolution strategy is dynamically adjusted according to the visual flow direction of the table content, when a cell spans multiple logical areas, the content distribution and the space weight are comprehensively evaluated, and a division scheme is selected, and the stress distribution of the internal data when the cell is split or merged is calculated, and a structure correlation strategy is formulated. The above scheme realizes accurate resolution and dimension reconstruction of the composite structure table through multi-modal feature collaborative analysis and dynamic topological optimization.

[0080] Embodiment 4: On the basis of embodiment 2, the process of the enhanced processing of the metadata of the content data provided by the present embodiment comprises the following steps:

[0081] S1031: The hierarchical pointer network is used to encode the table topological structure, the visual layout features are converted into spatial vector sequences through discrete cosine transformation, the projection relationship with the content domain is established, the font weighting coefficient of the title area, the cell merging mode and other 16 visual parameters are automatically identified by the positioning engine, and the spatial fingerprint is generated;

[0082] S1032: Develop a dynamic timestamp parser to extract the timing information of the spatial fingerprint from the following channels: revision log watermark embedded in the document, version hash value of the associated data stream, and timeline alignment of the external knowledge graph; eliminate timing conflicts through multiple hypothesis testing algorithms, and finally generate a confidence-weighted time interval representation;

[0083] S1033: Construct a heterogeneous source recognition engine based on fuzzy string matching, including parsing the layout features of footnote special characters, comparing the distribution probability of domain-specific terms in professional dictionaries, and extracting the meta-tag mode of the document container (such as PDF / HTML).

[0084] The working principle and beneficial effects of the above technical solution are: Firstly, the hierarchical pointer network is used to encode the table topological structure, the visual layout features are converted into spatial vector sequences through discrete cosine transformation, and the projection relationship with the content domain is established; the positioning engine automatically identifies 16 visual parameters such as the font weighting coefficient of the title area and the cell merging mode, and generates a spatial fingerprint; secondly, a dynamic timestamp parser is developed to extract the timing information of the spatial fingerprint from the following channels: revision log watermark embedded in the document, version hash value of the associated data stream, and timeline alignment of the external knowledge graph; eliminate timing conflicts through multiple hypothesis testing algorithms, and finally generate a confidence-weighted time interval representation; finally, a heterogeneous source recognition engine based on fuzzy string matching is constructed, including parsing the layout features of footnote special characters, comparing the distribution probability of domain-specific terms in professional dictionaries, and extracting the meta-tag mode of the document container (such as PDF / HTML). The above scheme systematically integrates spatial feature encoding, timing analysis engine, and cross-domain tracing mechanism to build a dynamic and adaptive metadata enhancement architecture; the spatial fingerprint and the time interval representation form a spatio-temporal joint vector through tensor concatenation, where the 16-dimensional visual parameters such as the font weighting coefficient and the confidence weight output by the timestamp parser are subjected to Hadamard product operation to generate a feature kernel with discrete-continuous hybrid characteristics; the feature kernel is subjected to orthogonalization processing through the domain term probability distribution matrix of the heterogeneous source recognition engine, and finally outputs a triple verification label carrying topological structure, version evolution, and source credibility. In the query stage, the system automatically activates the cross-modal attention mechanism of the feature kernel: the spatial vector sequence triggers approximate matching based on Manhattan distance, the time interval representation starts dynamic alignment of the sliding window, and the source feature Bloom filter performs fast pre-screening; the gradients of the three types of operation results are dynamically aggregated through the gated recurrent unit, so that the retrieval process meets the three constraint conditions of structural similarity, timing continuity, and source reliability. The feature drift detection module is introduced to maintain robustness: when the document container meta-tag mode deviates from the historical record by more than 3σ, the spatio-temporal joint vector recalibration process is automatically triggered, which includes version rollback based on revision log watermark, frequency domain feature update based on discrete cosine transformation, and recalculation of domain term distribution probability.

[0085] Embodiment 5: As shown in the embodiment 1, on the basis of the embodiment 1, the table data processing process of the mixed index and the joint retrieval provided by the embodiment of the application comprises the following steps: Figure 4

[0086] S201: For the structured table with a single table header, cell-level index decomposition is adopted, each row or key cell is mapped to an independent index unit, each index unit records its number, date, text, and row number and column number in the table, and maintains a mapping relationship with the overall structure of the table;

[0087] S202: The complete table structure is converted into a structured data format containing nested relationships, and the data level association therein is extracted; the membership relationship between the table header cell and the data cell is recorded in the index layer, and the dependence path of the underlying data on the high-level semantics is marked through the reverse pointer;

[0088] S203: Identify the adjacent text such as explanatory paragraphs and footnotes associated with the table, extract key information such as term explanation and unit definition; bind the relevant semantic information in the adjacent text to the index record in units of table cell or row data, and create a bidirectional index association.

[0089] The working principle and beneficial effects of the above technical solution are: firstly, for the structured table with a single table header, cell-level index decomposition is adopted, each row or key cell is mapped to an independent index unit, each index unit records its number, date, text, and row number and column number in the table, and maintains a mapping relationship with the overall structure of the table; secondly, the complete table structure is converted into a structured data format containing nested relationships, and the data level association therein is extracted; the membership relationship between the table header cell and the data cell is recorded in the index layer, and the dependence path of the underlying data on the high-level semantics is marked through the reverse pointer; finally, identify the adjacent text such as explanatory paragraphs and footnotes associated with the table, extract key information such as term explanation and unit definition; bind the relevant semantic information in the adjacent text to the index record in units of table cell or row data, and create a bidirectional index association. The table data processing process of the mixed index and the joint retrieval of the above solution substantially constructs a multi-level, strong semantic association dynamic index system, converts discrete technical components (coordinate index, hierarchical modeling, semantic binding) into a mutual feedback enhancement loop, so that any single point improvement (such as more detailed adjacent text analysis) can be used to improve the overall retrieval efficiency through system cooperation.

[0090] Embodiment 6: On the basis of the embodiment 5, the process of creating a bidirectional index association provided by the embodiment of the application comprises the following steps:

[0091] ​S2031: Based on the generated structured data hierarchy, identify the cells in the table that have a semantic pivot role (such as table headers, statistical indicator name cells across rows and columns); determine the high-level semantic scope that each data cell needs to be bound to through the reverse pointer marked dependency path (such as the monthly data column under the "annual revenue" cell); after processing the adjacent text, complete the normalization marking of terms and units;

[0092] S2032: Automatically divide the semantic influence domain according to the table layout features; when the column relationship is the main factor in the vertical direction, bind the footnote translation information in the 3 rows below the table header to the column; for horizontal cross-page tables, capture the continuity explanation text before and after the page break; use a dynamic sliding window mechanism to form an inverse weighting between the cell position coordinates and the physical distance of the text block;

[0093] S2033: Establish a synonymous hash between the original content of the cell and the standard expression in the structured index; convert the unit definition extracted from the adjacent text into a constraint condition expression and attach it to the metadata layer of the corresponding data cell; generate a semantic tree through the table header membership relationship, so that the footnote information can drill down to the end cell along the tree path, forming a bidirectional addressing capability of cells, standard terms, and explanatory text.

[0094] The working principle and beneficial effects of the above technical solution are: first, based on the generated structured data hierarchy, identify the cells in the table that have a semantic pivot role (such as table headers, statistical indicator name cells across rows and columns); determine the high-level semantic scope that each data cell needs to be bound to through the reverse pointer marked dependency path (such as the monthly data column under the "annual revenue" cell); after processing the adjacent text, complete the normalization marking of terms and units; second, automatically divide the semantic influence domain according to the table layout features; when the column relationship is the main factor in the vertical direction, bind the footnote translation information in the 3 rows below the table header to the column; for horizontal cross-page tables, capture the continuity explanation text before and after the page break; use a dynamic sliding window mechanism to form an inverse weighting between the cell position coordinates and the physical distance of the text block; finally, establish a synonymous hash between the original content of the cell and the standard expression in the structured index; convert the unit definition extracted from the adjacent text into a constraint condition expression and attach it to the metadata layer of the corresponding data cell; generate a semantic tree through the table header membership relationship, so that the footnote information can drill down to the end cell along the tree path, forming a bidirectional addressing capability of cells, standard terms, and explanatory text. The above scheme constructs a three-order verification chain of geometric position correlation, semantic category matching, and logical constraint effectiveness, forming a reversible closed loop mapping between the physical structure of the table data and the business context logic. The cell row and column coordinates serve as the bottom positioning reference, the structured hierarchy relationship serves as the semantic relay function, and the adjacent text provides dynamic verification rules; through progressive conditional constraints, the system achieves collaborative operation and finally achieves bidirectional traceability of the index record and the explanation system.

[0095] Embodiment 7: Based on Embodiment 1, the process of vector retrieval and keyword retrieval provided by the embodiment of the application comprises the following steps:

[0096] S204: The key-value combination analysis logic identifies the core entity and attribute requirement in the free-text query input by the user and converts them into a structured key-value pair; automatically constructs a query logic tree based on the key-value combination and injects it into the index system to perform directional retrieval;

[0097] S205: If ambiguous terms are detected in the key-value analysis stage, activate the context joint index, infer the corresponding time interval by searching adjacent text, supplement semantics to the query condition, and generate an extended key-value combination;

[0098] S206: Perform parallel retrieval using the composite index structure of the hybrid index and the context joint index, directly call the point search mechanism of the fine-grained index, and return the matched cell or row data; locate the sub-tree structure that meets the conditions through hierarchical traversal of the nested index.

[0099] The working principle and beneficial effects of the above technical solution are as follows: first, the key-value combination analysis logic identifies the core entity and attribute requirement in the free-text query input by the user and converts them into a structured key-value pair; automatically constructs a query logic tree based on the key-value combination and injects it into the index system to perform directional retrieval; second, if ambiguous terms are detected in the key-value analysis stage, activate the context joint index, infer the corresponding time interval by searching adjacent text, supplement semantics to the query condition, and generate an extended key-value combination; finally, perform parallel retrieval using the composite index structure of the hybrid index and the context joint index, directly call the point search mechanism of the fine-grained index, and return the matched cell or row data; locate the sub-tree structure that meets the conditions through hierarchical traversal of the nested index. The above scheme realizes efficient retrieval through the coupling design of hybrid index and joint retrieval, regardless of whether the original table is sparse (single-level table header) or complex (multi-level nested), and can complete efficient retrieval through a unified interface; the context binding mechanism ensures that the retrieval result does not deviate from its original semantic environment (such as avoiding misinterpretation of numbers that deviate from the unit explanation); the index strategy optimizes itself according to the query mode, avoiding local hotspots or resource waste caused by fixed architecture.

[0100] Embodiment 8: Based on Embodiment 7, the process of generating an extended key-value combination provided by the embodiment of the application comprises the following steps:

[0101] S2051: Based on the key-value pair structure in the parsed free text query, detect the fuzzy descriptions contained in the attribute requirement field and locate the corresponding logical node in the index system; extract the associated semantic fragments of the node in the adjacent text according to the activated context composite index to form a candidate supplementary information pool; rely on the entity type marker in the structured key-value pair as a filtering condition to exclude mismatched context fragments.

[0102] S2052: Semantic quantification transformation is performed on the identified fuzzy terms. Time-related terms are mapped to closed interval value ranges by the reference year in the retrieval context. Degree-related terms are converted into numerical threshold conditions by combining the statistical description of adjacent text. The nested hierarchical structure of the hybrid index is used to backtrack along the membership path of the table header and data to calibrate the consistency of the order of magnitude and units.

[0103] S2053: The original key-value pairs are evolved according to the three rules, directly replacing the fuzzy terms with specific intervals; derivative constraints are added to the original precise key-value pairs; and supplementary conditions are injected into the associated sub-index fields through the subtree positioning mechanism to generate new key-value combinations with complete context constraints.

[0104] The working principle and beneficial effects of the above technical solution are as follows: First, based on the key-value pair structure in the parsed free text query, this embodiment detects the fuzzy expressions contained in the attribute requirement field and locates the corresponding logical node in the index system; according to the activated context composite index, it extracts the associated semantic fragments of the node in the adjacent text to form a candidate supplementary information pool; relying on the entity type marker in the structured key-value pair as a filtering condition, it excludes mismatched context fragments; secondly, it performs semantic quantization transformation on the identified fuzzy terms, mapping time-related expressions to a closed interval value range through the benchmark year in the retrieval context; degree-related expressions are converted into numerical threshold conditions by combining the statistical description of adjacent text; using the nested hierarchical structure of the hybrid index, it backtracks along the membership path of the table header-data to calibrate the consistency of the magnitude unit; finally, it evolves the original key-value pairs according to the three rules, directly replacing the fuzzy items with specific intervals; it adds derivative constraints to the original precise key values; and through the subtree positioning mechanism, it injects the supplementary conditions into the associated sub-index domain to generate a new key-value combination with complete context constraints. The above scheme forms a chain of processing, including fuzzy recognition, context extraction, quantization transformation, and condition fusion, so that the expanded key-value combination inherits the original query intent and integrates the contextual information of the table itself. Among them, the structured key-value pairs serve as the control skeleton, the contextual composite index provides semantic filling material, and the hierarchical characteristics of the hybrid index undertake the function of ensuring logical integrity.

[0105] Example 9: As Figure 5 As shown, based on Example 1, the process of adjusting the hybrid index and contextual composite index to a row, column, or sub-table provided in this embodiment of the invention includes the following steps:

[0106] S301: Quantitative analysis of the distribution of blank areas in the sparse table based on the joint feedback results of vector retrieval and keyword retrieval; when the proportion of continuous longitudinal blank cells exceeds the threshold, the column index mode is triggered, and at this time the data type of each column and the semantic of the adjacent file header are established horizontally joint index; if the horizontal blank cells are dense, switch to the row index mode, and the semantic correlation degree of the key value of the row and the data of the upper and lower rows is used as the weight factor; the blank area is composed of multiple blank cells;

[0107] S302: Extract the topological features of the sub-table according to the table header hierarchical relationship processed in step; when the user query key value combination involves multi-layer association, automatically decompose the original single-level index into a tree-like sub-table index network: the upper index node stores the sub-table position pointer, and the lower node retains the numerical range statistical features of the specific cell;

[0108] S303: Use the generated lightweight calculation results as a verification data set to implement three-stage optimization by comparing the delay-accuracy surface of the test case:

[0109] First-level optimization: For high-frequency query-related index cells (such as the "net profit" column of the financial statement), increase the dimension weight of its vector retrieval;

[0110] Second-level optimization: When there is a cross-file reference conflict in the context joint index (such as the difference in measurement units of the same table header in different files), trigger the standardization compensation mechanism to add unit conversion coefficient anchor points at the index layer;

[0111] Third-level optimization: Based on the heat map distribution of the query results, implement compression and reorganization of sub-table indexes below the response threshold, and reduce them to Boolean combination expression of row / column index.

[0112] The working principle and beneficial effects of the technical solution are as follows: first, based on the joint feedback result of vector retrieval and keyword retrieval, the distribution of blank areas of the sparse table is quantitatively analyzed; when it is detected that the proportion of continuous longitudinal blank cells exceeds a threshold value, the column index mode is triggered, at this time, the data type of each column and the semantic of the header of the adjacent file are established horizontally combined index; if the horizontal blank cells are dense, the row index mode is switched to, the semantic correlation degree of the key value of the row and the data of the upper and lower rows is taken as a weight factor; secondly, according to the table header hierarchical relationship processed in the step, the topological features of the sub-table are extracted; when the user query key value combination involves multi-layer association, the original single-level index is automatically disassembled into a tree-like sub-table index network: the upper index node stores the sub-table position pointer, and the lower node retains the numerical range statistical characteristics of the specific cell; finally, the generated lightweight calculation result is used as a verification data set, and by comparing the delay-accuracy surface of the test case, three-stage optimization is implemented: first-stage optimization: for the index unit involved in high-frequency queries (such as the "net profit" column of the financial report), the weight of the vector retrieval dimension is improved; second-stage optimization: when there is a cross-file reference conflict in the context combined index (such as the difference in the unit of measurement of the same table header in different files), the standardization compensation mechanism is triggered, and the unit conversion coefficient anchor point is added to the index layer; third-stage optimization: based on the heat map distribution of the query result, the sub-table index below the response threshold is compressed and reorganized, and it is reduced to a Boolean combination expression of row / column index. The dynamic selection of the index granularity of the above-mentioned scheme directly determines the organization form of the combined index, and the real-time feedback data reversely constrains the strategy of index fusion; the whole process realizes progressive enhancement through quantitative indicators (blank cell threshold, response delay, etc.), which meets the core demand of improving data processing efficiency through structural innovation.

[0113] In the embodiments 1-9, the RAG-based table data processing system provided by the embodiments of the present application comprises: Figure 6

[0114] The analysis and conversion module is responsible for acquiring files with structured characteristics and their electronic file formats, acquiring the table analysis path in the files from the pre-stored analysis program according to the electronic file format; and performing enhancement processing, structured processing, semantic processing and standardization processing on the content data in the table to obtain processed content data; wherein the content data includes metadata, table headers, data in each row and digital dates of the table.

[0115] The retrieval adaptation module is responsible for receiving the processed content data, establishing a mixed index suitable for different table structures of sparse tables and multi-level tables and a context combined index of the content data in the table and the merged index of the adjacent files; acquiring user query content, converting it into a key value combination, and performing vector retrieval and keyword retrieval;

[0116] ​The generation integration module is responsible for dynamic template filling according to the results of key-value combination, vector retrieval and keyword retrieval, instructing the data fine-tuning model to guide the generation of instructions, and simultaneously generating lightweight calculation results; according to test cases, combining query delay and accuracy, adjusting the mixed index and the context joint index to be a row, a column or a sub-table.

[0117] The working principle and beneficial effects of the technical solution are as follows: the analysis conversion module of the embodiment acquires a file with a structured characteristic and an electronic file format thereof, acquires a table analysis path in the file from a pre-stored analysis program according to the electronic file format; performs enhancement processing, structured processing, semantic processing and standardization processing on content data in the table to obtain processed content data; the content data includes metadata, table headers, each row of data and digital dates of the table; the retrieval adaptation module receives the processed content data, establishes a mixed index suitable for different table structures of sparse tables and multi-level tables and a context joint index of the content data in the table and a merged index of adjacent files; acquires user query content, converts the user query content into a key-value combination, performs vector retrieval and keyword retrieval; the generation integration module performs dynamic template filling according to the results of the key-value combination, the vector retrieval and the keyword retrieval, instructs the data fine-tuning model to guide the generation of instructions, and simultaneously generates lightweight calculation results; according to test cases, combining query delay and accuracy, adjusting the mixed index and the context joint index to be a row, a column or a sub-table. The RAG-based table data processing system of the above scheme constructs a complete table data processing pipeline through the cooperative operation of the analysis conversion module, the retrieval adaptation module and the generation integration module. The analysis conversion module realizes the unified structured representation of cross-format table data, providing standardized input for downstream processing; the retrieval adaptation module establishes a mixed index system based on structured data, realizing unified support for multi-modal query methods; the generation integration module completes the explainability conversion of the query results through the template reorganization and lightweight calculation of the retrieval results. The embodiment establishes a mapping channel from the original table to the final result through feature conversion and data processing at each link; through the cooperation of the mixed index and the dynamic template, the contradiction between query flexibility and result accuracy in the traditional table processing method is solved; through the index adjustment mechanism based on performance feedback, the adaptive balance between processing quality and response efficiency of the system is realized, and finally an intelligent closed-loop system covering the whole life cycle of data processing is formed.

[0118] The application scenarios of the embodiment include: financial and business analysis, medical health, scientific research and academic research, enterprise operation and supply chain, customer service and sales, and law and compliance, etc.

[0119] Among them, financial and business analysis includes financial statement analysis and risk monitoring; financial statement analysis includes automatically extracting table data such as revenue, cost and profit in enterprise financial statements; risk monitoring includes real-time retrieval of loan application forms, risk assessment and generation of early warning prompts.

[0120] Medical health includes patient medical record management and scientific research data analysis; patient medical record management includes patient condition analysis combined with electronic medical record tables (check indicators, medication records); scientific research data analysis includes extracting drug dosage and efficacy relationship from clinical trial tables to generate statistical conclusions.

[0121] Scientific research and academic research include experimental data organization and literature review support; experimental data organization includes analyzing experimental record tables (temperature, pressure, reaction time); literature review support includes aggregating data across paper tables to generate field trend analysis.

[0122] Enterprise operation and supply chain include inventory management and supply chain optimization; inventory management includes generating replenishment suggestions based on inventory tables (SKU, quantity, shelf life); supply chain optimization includes analyzing logistics cost tables to recommend the optimal supplier combination.

[0123] Customer service and sales include CRM data query and sales report generation; CRM data query includes quickly retrieving customer information tables to answer customer questions; sales report generation includes automatically generating visual summary reports based on sales performance tables.

[0124] Legal and compliance include contract clause retrieval and regulation comparison; contract clause retrieval includes locating "breach of contract liability" related clauses from contract clause tables to generate compliance risk summaries; regulation comparison includes analyzing tables in multi-country legal documents to output difference comparison lists.

[0125] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the equivalent technology of the present application, the present application also intends to include these modifications and variations.

Claims

1. A RAG-based table data processing method, characterized by, Comprising the following steps: Obtain user query content, convert to key-value combination, perform vector retrieval and keyword retrieval; according to the results of key-value combination, vector retrieval and keyword retrieval, perform dynamic template filling, instruct data fine-tuning model to guide instruction generation; According to the test case, combined with the query delay and accuracy, adjust the mixed index and the context combined index to the row, column or sub-table; based on the joint feedback results of vector retrieval and keyword retrieval, quantitatively analyze the distribution of blank areas of sparse table; according to the processed table header hierarchical relationship, extract the topological features of the sub-table; When the user query key-value combination involves multi-layer association, automatically decompose the original single-level index into a tree-like sub-table index network, use the generated lightweight calculation results as the verification data set, and implement three-stage optimization by comparing the delay-accuracy surface of the test case.

2. The RAG-based table data processing method of claim 1, wherein, The content data in the table is enhanced, structured, semantically processed and standardized to obtain processed content data.

3. The RAG-based table data processing method of claim 2, wherein, The process of obtaining processed content data comprises the following steps: Create a repository for storing the parsing program; When a file with structural characteristics is obtained, issue an instruction to trigger the parsing program, and sequentially confirm the electronic file format and the parsing method according to the identification process of the parsing program.

4. The RAG-based table data processing method of claim 3, wherein, When the electronic file format is confirmed, parse according to the corresponding interpretation method; take the electronic file format as the starting node and the parsing method as the end node to form the table parsing path in the file.

5. The RAG-based table data processing method of claim 4, wherein, The electronic file format is the file extension, and the file includes word, excel and pdf.

6. The RAG-based table data processing method of claim 3, wherein, After forming the table parsing path in the file, obtain the metadata, table header, each row of data and numerical date content data in the table, and perform enhancement, structuring, semantic processing and standardization on different content data.

7. The RAG-based table data processing method of claim 3, wherein, The parsing program includes electronic file format and corresponding parsing method, and files with structural characteristics that trigger the parsing program.

8. A RAG-based table data processing system, characterized by, Comprise: The parsing conversion module is responsible for enhancing, structuring, semantically processing and standardizing the content data in the table to obtain processed content data; The retrieval adaptation module is responsible for receiving the processed content data, establishing a mixed index and a context combined index; obtaining user query content, converting to key-value combination, performing vector retrieval and keyword retrieval; The generation integration module is responsible for dynamic template filling according to the results of key-value combination, vector retrieval and keyword retrieval, instructing data fine-tuning model to guide instruction generation; according to the test case, combined with the query delay and accuracy, adjusting the mixed index and the context combined index to the row, column or sub-table; Based on the joint feedback results of vector retrieval and keyword retrieval, quantitatively analyze the distribution of blank areas of sparse table; according to the processed table header hierarchical relationship, extract the topological features of the sub-table; When the user query key-value combination involves multi-layer association, automatically decompose the original single-level index into a tree-like sub-table index network, use the generated lightweight calculation results as the verification data set, and implement three-stage optimization by comparing the delay-accuracy surface of the test case.

Images