A table processing method and device based on an adaptive strategy

Abstract

This invention belongs to the field of data management and discloses a table processing method and apparatus based on an adaptive strategy. The method includes: acquiring table data and generating structured data based on the table data; determining the characteristics of the structured data and selecting a corresponding processing strategy based on the characteristics; and processing the structured data based on the processing strategy to obtain a corresponding metadata model. This invention dynamically selects an appropriate processing strategy based on the characteristics of the structured data, transforming the table data into a unified standard metadata model. This adaptive processing method can flexibly handle various types of tables without requiring customized processing for specific table structures, thus improving the efficiency of table processing. Simultaneously, this process reduces the problems of structure recognition errors and semantic information loss when dealing with complex tables, improving the accuracy of table processing and consequently improving the retrieval accuracy and recall rate of the knowledge base.

Description

Technical Field

[0001] This invention relates to the field of data management, and more specifically, to a table processing method and apparatus based on an adaptive strategy. Background Technology

[0002] With the advancement of digital transformation across industries, building intelligent knowledge bases using Retrieval-augmented Generation (RAG) technology alongside document technology has become mainstream. However, technical documents often contain a large number of complex and heterogeneous tables, presented in vastly different ways and lacking a unified structural standard. Traditional Optical Character Recognition (OCR) technology and general document parsing tools often struggle to accurately identify the table structure when processing these heterogeneous tables, leading to the loss of semantic information. This means that when building intelligent knowledge bases based on table data, effective retrieval and understanding are impossible, severely impacting information utilization efficiency and the retrieval accuracy of the knowledge base.

[0003] Therefore, improving the efficiency and accuracy of table processing is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] To address one or more deficiencies in existing technologies, this invention provides a table processing method and apparatus based on an adaptive strategy.

[0005] A table processing method based on an adaptive strategy includes: Obtain tabular data and generate structured data based on the tabular data; Determine the characteristics of the structured data, and select the corresponding processing strategy based on the characteristics; The structured data is processed based on the aforementioned processing strategy to obtain the corresponding metadata model.

[0006] Optionally, determining the characteristics of the structured data includes: The complexity of the table is determined based on the span attribute of the structured data; The structured data is matched using a first regular expression, and the interactive markers of the table are determined based on the first matching result; the first regular expression is used to match preset interactive keywords. The structured data is matched using a second regular expression, and the remaining item markers of the table are determined based on the second matching result; the second regular expression is used to match preset remaining item keywords. The global semantic entropy value of the table is determined based on the text repetition frequency of the structured data; The blank rate of the table is determined based on the number of blank cells in the structured data. The complexity, the interaction marker, the remainder marker, the global semantic entropy value, and the blank rate are determined as features of the structured data.

[0007] Optionally, the step of selecting the corresponding processing strategy based on the features includes: Determine whether the complexity is high. If so, then choose the complex structure processing strategy; If not, then determine whether the remaining item flag is positive; If the remaining item is marked as positive, then the implicit entity augmentation processing strategy is selected; If the remainder is marked as negative, then determine whether the global semantic entropy value is less than a first preset threshold. If the global semantic entropy value is less than the first preset threshold, then a redundant information compression processing strategy is selected. If the global semantic entropy value is greater than or equal to the first preset threshold, then determine whether the blank rate is greater than the second preset threshold; If the blank rate is greater than the second preset threshold, then the interactive form filling template processing strategy is selected; If the blank rate is less than or equal to the second preset threshold, then a general processing strategy is selected.

[0008] Optionally, when the processing strategy is the complex structured processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Construct a virtual coordinate matrix and map the structured data to the virtual coordinate matrix; Copy the text value of the merged cell in the structured data and fill it into each virtual coordinate point within the logical placeholder field of the merged cell; For each virtual coordinate point in the virtual coordinate matrix, extract all the headers of the virtual coordinate point, and concatenate all the headers with the text value of the virtual coordinate point to obtain the full path semantic chain of the virtual coordinate point; The virtual coordinates of the virtual coordinate points and the full path semantic chain are filled into the first metadata template to obtain the corresponding metadata model; the first metadata template is the metadata template corresponding to the complex structure processing strategy.

[0009] Optionally, when the processing strategy is the implicit entity augmentation processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: The structured data is matched using the second regular expression to identify target cells and determine the header attribute of the column to which each target cell belongs; the target cell contains the remaining keywords; Construct an explicit entity collection based on the text values ​​in non-target cells; Obtain the full set of entities under the header attribute from the preset knowledge base, and determine the implicit entity set by the difference between the full set of entities and the displayed entity set. The implicit entity set is converted into natural language description text, and the natural language description text is injected as extended data into the data object of the target cell; The physical coordinates of the target cell and the extended data are filled into the second metadata template to obtain the corresponding metadata model; the second metadata template is the metadata template corresponding to the implicit entity expansion processing strategy.

[0010] Optionally, when the processing strategy is the redundant information compression processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Clustering is performed on the data regions in the structured data to obtain multiple data clusters; Perform core feature extraction processing on each data cluster to obtain the core business value of each data cluster; Generate corresponding summary text based on the core business values; The physical coordinates covered by each data cluster and the summary text are filled into the third metadata template to obtain the corresponding metadata model; the third metadata template is the metadata template corresponding to the redundant information compression processing strategy.

[0011] Optionally, when the processing strategy is the interactive form-filling template-based processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Based on the content characteristics of each cell in the structured data, the cells are divided into labels or slots; Establish a mapping relationship between the slot and the nearest tag to form a tag-slot pair; Semantic parsing is performed on the tag-slot pairs to extract the filling constraints; The physical coordinates of the tag-slot pair and the filling constraints are filled into the fourth metadata template to obtain the corresponding metadata model; the fourth metadata template is the metadata template corresponding to the interactive filling template type processing strategy.

[0012] Optionally, the metadata model includes a natural language description field; After processing the structured data based on the processing strategy to obtain the corresponding metadata model, the method further includes: The natural language description fields are converted into vector features; An index entry is constructed in a pre-defined vector database; the index entry includes a vector index area and a scalar payload area; The metadata model is serialized, and the processing result is stored in the scalar payload area of ​​the index entry. The vector features corresponding to the metadata model are stored in the vector index area of ​​the index entry.

[0013] A table processing system based on an adaptive strategy includes: A preprocessing module is used to acquire tabular data and generate structured data based on the tabular data; The feature extraction and decision-making module is used to determine the features of the structured data and select the corresponding processing strategy based on the features; The data processing module is used to process the structured data based on the processing strategy to obtain the corresponding metadata model.

[0014] An electronic device, comprising: A processor and a memory, the memory being used to store at least one instruction, which, when loaded and executed by the processor, implements the table processing method based on an adaptive strategy as described in any of the preceding embodiments.

[0015] The table processing method based on an adaptive strategy provided in this invention includes: acquiring table data and generating structured data based on the table data; determining the features of the structured data and selecting a corresponding processing strategy based on the features; and processing the structured data based on the processing strategy to obtain a corresponding metadata model. This invention dynamically selects an appropriate processing strategy based on the features of the structured data, transforming the table data into a unified standard metadata model. This adaptive processing method can flexibly handle various types of tables without requiring customized processing for specific table structures, thus improving the efficiency of table processing. Simultaneously, this process reduces the problems of structure recognition errors and semantic information loss when dealing with complex tables, improving the accuracy of table processing and consequently improving the retrieval accuracy and recall rate of the knowledge base. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a table processing device based on an adaptive strategy provided in an embodiment of the present invention; Figure 2 for Figure 1 A flowchart illustrating an actual manifestation of step S02 in a table processing method based on an adaptive strategy; Figure 3 for Figure 1 A flowchart illustrating another practical manifestation of step S02 in a table processing method based on an adaptive strategy. Figure 4 for Figure 1 A flowchart illustrating the first practical manifestation of step S03 in a table processing method based on an adaptive strategy. Figure 5 This is a schematic diagram of a complex structured table provided in an embodiment of the present invention; Figure 6 for Figure 1 A flowchart illustrating the second practical manifestation of step S03 in a table processing method based on an adaptive strategy. Figure 7 This is a schematic diagram of an implicit entity extended table provided in an embodiment of the present invention; Figure 8 for Figure 1 A flowchart illustrating the third practical manifestation of step S03 in a table processing method based on an adaptive strategy. Figure 9 This is a schematic diagram of a redundant information compressed table provided in an embodiment of the present invention; Figure 10 for Figure 1 A flowchart illustrating the fourth practical manifestation of step S03 in a table processing method based on an adaptive strategy. Figure 11 This is a schematic diagram of an interactive form-filling template provided in an embodiment of the present invention; Figure 12 A flowchart illustrating another table processing method based on an adaptive strategy provided in an embodiment of the present invention. Figure 13This is a schematic diagram of the structure of a table processing device based on an adaptive strategy provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of another table processing device based on an adaptive strategy provided in an embodiment of the present invention. Detailed Implementation

[0018] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] With the advancement of digital transformation across industries, building intelligent knowledge bases using search-enhanced generation technologies and plug-in document technologies has become mainstream. However, technical documents often contain a large number of complex and heterogeneous tables, presented in vastly different ways and lacking a unified structural standard. Traditional OCR technologies and general document parsing tools often struggle to accurately identify the table structure when processing these heterogeneous tables, leading to the loss of semantic information. This means that when building intelligent knowledge bases based on table data, effective retrieval and understanding are impossible, severely impacting information utilization efficiency and the accuracy of knowledge base retrieval.

[0023] Therefore, the present invention provides a table processing method and apparatus based on an adaptive strategy to solve the above problems.

[0024] Please refer to Figure 1 The flowchart below shows a table processing method based on an adaptive strategy provided by an embodiment of the present invention, which includes the following steps: Step S01: Obtain tabular data and generate structured data based on the tabular data.

[0025] In this embodiment, acquiring tabular data and generating structured data based on it refers to transforming raw, unstructured tabular data into structured data that a computer can recognize and process. Raw tabular data is typically a PDF, image, or other formatted document containing text, images, and layout information. Structured data refers to organizing the data in a table according to attributes such as rows (tr), columns (td), row spans (rowspan), and column spans (colspan), forming a structured data tree that can be accessed and manipulated by a program.

[0026] Since unstructured data cannot be directly analyzed and utilized, it needs to be transformed into a structured form for subsequent processing. Tables in technical documents often have complex layouts, making direct semantic analysis and knowledge extraction difficult. Therefore, visual layout analysis and structure transformation are necessary to extract the data content and structural information, laying the foundation for subsequent semantic understanding and vectorization.

[0027] In some embodiments, the acquisition of tabular data mentioned in step S01, and the generation of structured data based on the tabular data, can specifically be as follows: Using primitive detection technology, table borders and table dividing lines are identified, non-table areas such as headers and footers are removed, and the physical boundaries of the table are determined. Use OCR technology to extract the text content within the table and transform the visual table into a basic HTML structure tree containing row, column, row span, and column span attributes.

[0028] Step S02: Determine the characteristics of the structured data and select the corresponding processing strategy based on the characteristics.

[0029] In this embodiment, features are extracted from the generated structured data, and an appropriate processing strategy is selected based on the extracted features. The features of the structured data may include blank rate, structural complexity, interactive markers, and global semantic entropy value. The processing strategy refers to using different processing methods for different types of tables.

[0030] Because technical documents contain a variety of table types, a single processing strategy cannot guarantee effective results. By extracting features and selecting appropriate strategies, more effective processing methods can be adopted for different types of tables. This avoids the waste of resources caused by complex processing of simple tables, while ensuring that complex tables are processed accurately, thereby improving overall processing efficiency and quality.

[0031] The specific implementation of step S02 can be referred to the embodiments shown in subsequent steps S11 to S16 and steps S31 to S39, which will not be repeated here.

[0032] Step S03: Process the structured data based on the processing strategy to obtain the corresponding metadata model.

[0033] In this embodiment, structured data is processed according to the selected processing strategy to ultimately generate a unified metadata model. The metadata model is a structured representation obtained by abstracting and summarizing tabular data. It includes structural information, semantic information, business information, and interaction information, and can be stored in JSON, XML, or other formats.

[0034] This embodiment converts the processing results of different types of tables into a unified JSON metadata model, which facilitates subsequent knowledge integration and retrieval. The unified metadata model reduces the adaptation costs for downstream systems (such as intelligent knowledge bases), enabling unified management and utilization of different types of table data.

[0035] The specific implementation of step S03 can be referred to the embodiments shown in subsequent steps S41 to S44, S51 to S55, S61 to S64, and S71 to S74, which will not be repeated here.

[0036] Based on the above technical solution, the table processing method based on adaptive strategies provided in this invention involves acquiring table data and generating structured data from it; determining the features of the structured data and selecting corresponding processing strategies based on these features; and processing the structured data using the processing strategies to obtain a corresponding metadata model. This invention dynamically selects appropriate processing strategies based on the features of the structured data, transforming table data into a unified standard metadata model. This adaptive processing method can flexibly handle various types of tables without requiring customized processing for specific table structures, thus improving the efficiency of table processing. Simultaneously, this process reduces structural recognition errors and semantic information loss when dealing with complex tables, improving the accuracy of table processing and consequently enhancing the retrieval accuracy and recall rate of the knowledge base.

[0037] Based on the above embodiments, in some embodiments, in order to improve the efficiency and accuracy of table processing and to select the most suitable processing strategy according to the actual situation of the table, the type and characteristics of the table can be understood through multi-dimensional feature extraction, so as to ensure that the selection of subsequent processing strategies is more accurate and effective.

[0038] Please refer to Figure 2 ,for Figure 1 The flowchart illustrates a practical manifestation of step S02 in a table processing method based on an adaptive strategy. Step S02, which involves determining the characteristics of structured data, can specifically include, for example... Figure 2 The steps shown are as follows: Step S11: Determine the complexity of the table based on the span attribute of the structured data.

[0039] In this embodiment, the span attribute describes the number of rows and columns a cell occupies in the table. Complex tables typically contain multiple cells with span attributes, and these cells may be nested in multiple layers. The complexity of the table directly affects the difficulty of subsequent processing. Tables with many span attributes usually have complex nested structures, which can easily lead to data breaks or missing information between the data and the table header, requiring more complex algorithms to parse correctly. By evaluating the complexity of the table, a basis can be provided for selecting subsequent processing strategies, allowing for the selection of more refined processing strategies.

[0040] In some embodiments, the row span and column span attribute values ​​of each cell can be counted by traversing the HTML structure tree, and a comprehensive complexity score can be calculated. If the table has cells that span multiple rows (Rowspan>1) or multiple columns (Colspan>1), and the table header hierarchy depth is greater than or equal to 2, it is marked as high complexity; otherwise, it is marked as low complexity.

[0041] Step S12: Use the first regular expression to match the structured data, and determine the interactive markers of the table based on the first matching result.

[0042] The first regular expression is used to match preset interactive keywords.

[0043] In this embodiment, the interaction marker is used to identify whether the table contains elements requiring user interaction, such as required fields or signature areas. A predefined first regular expression is used to scan the text in the table to find preset interaction keywords, such as "required," "signature," and "date." If a preset interaction keyword is matched, the interaction marker is set to positive; otherwise, it is set to negative.

[0044] Some forms require users to fill out or confirm them, such as application forms and approval forms. By identifying interaction keywords, we can determine whether a form belongs to this category, thus providing a basis for subsequently selecting the appropriate interactive form-filling template processing strategy.

[0045] Step S13: Use the second regular expression to match the structured data, and determine the remaining item markers of the table based on the second matching result.

[0046] The second regular expression is used to match preset keywords.

[0047] In this embodiment, the remainder marker is used to identify whether words such as "other" or "remaining" exist in the table, indicating the presence of data that is not explicitly listed. A predefined second regular expression is used to scan the text in the structured data to find preset remainder keywords. If a preset remainder keyword is matched, the remainder marker is set to positive; otherwise, it is set to negative.

[0048] Because some table data is not listed in its entirety, but instead uses terms like "other" or "remaining" to indicate data not listed, and intelligent knowledge bases cannot recognize this type of table data, the completeness of the information is affected. Identifying these remaining item tags can guide subsequent implicit entity expansion operations to supplement the missing information.

[0049] Step S14: Determine the global semantic entropy value of the table based on the text repetition frequency of the structured data.

[0050] In this embodiment, the degree of semantic information disorder in the table data is calculated by statistically analyzing the frequency of repeated occurrences of text content in the table data area; that is, the global semantic entropy value. The higher the global semantic entropy value, the more random and less repetitive the information in the table, and vice versa.

[0051] Global semantic entropy reflects the redundancy level of tabular data. Tables with high redundancy have lower information content and can be processed using compression strategies to improve storage efficiency and retrieval speed.

[0052] In some embodiments, the step S14, which involves determining the global semantic entropy value of the table based on the text repetition frequency of the structured data, may specifically include the following steps: Step S21: Construct a full sample set based on structured data.

[0053] In this embodiment, the header area and data area of ​​the structured data are identified, and only the text content of all non-empty cells in the data area is extracted to construct the full sample set D.

[0054] Step S22: Perform frequency statistics based on the full sample set.

[0055] In this embodiment, the entire sample set D is deduplicated to obtain a set U = {u1, u2, ..., uk} consisting of k unique text values. Simultaneously, the frequency fi of each unique text value ui in D is counted.

[0056] Step S23: Calculate the probability of each unique text value appearing in the full sample set.

[0057] In this embodiment, the probability can be calculated using formula (1): (1) Where pi is the probability of the i-th type of text value ui appearing in the data area, fi is the frequency of the unique text value ui appearing in D, and i = 1, 2, ..., k, k is the total number of non-repeating text value categories in the table data area, and N is the total number of cells contained in D.

[0058] Step S24: Calculate the global semantic entropy value based on probability.

[0059] In this embodiment, the global semantic entropy value can be calculated using formula (2): (2) Where H(D) is the global semantic entropy value to be solved, in bits.

[0060] Step S15: Determine the blank rate of the table based on the number of blank cells in the structured data.

[0061] In this embodiment, a blank cell is a cell in the data area that is empty, contains only spaces, or contains only placeholders (such as " / ", "-", etc.). Determining the blank rate of a table based on the number of blank cells in the structured data involves traversing each cell in the table's data area and determining whether it is a blank cell. The number of blank cells is counted, and the proportion of blank cells to the total number of cells in the table is calculated to obtain the blank rate. The blank rate reflects the completeness of the table data.

[0062] Step S16: Complexity, interaction markers, remainder markers, global semantic entropy value, and blank rate are determined as features of structured data.

[0063] Since individual features may not be sufficient to fully describe the characteristics of a table, in this embodiment, by determining complexity, interaction markers, remainder markers, global semantic entropy values, and blank rates as features of structured data, the characteristics of the table can be described more comprehensively and accurately, thereby providing a more reliable basis for subsequent strategy selection.

[0064] Furthermore, the complexity score, interaction markers, remainder markers, global semantic entropy value, and blank rate value can be combined into a vector containing multiple elements. This vector can then be used as a feature of the structured data for subsequent processing strategy selection.

[0065] Based on the above embodiments, in some embodiments, in order to improve the intelligence and efficiency of the table processing flow, a series of judgment conditions can be used to adaptively select the most suitable processing strategy based on the table's complexity, whether there are any duplicate items, semantic entropy value, blank rate, and other characteristics, so as to avoid using a uniform processing method for all tables and maximize the efficiency of table processing and the accuracy of the results.

[0066] Please refer to Figure 3 ,for Figure 1 The flowchart illustrates another practical manifestation of step S02 in the provided adaptive strategy-based table processing method. The feature selection-based processing strategy mentioned in step S02 can specifically include, for example... Figure 3 The steps shown are as follows: Step S31: Determine whether the complexity is high.

[0067] If yes, proceed to step S32; otherwise, proceed to step S33.

[0068] In this embodiment, the complexity is determined based on the previously calculated complexity to determine whether it is high complexity. If it is high complexity, the processing flow of the complex structure processing strategy is triggered.

[0069] Step S32: Select a complex structure processing strategy.

[0070] Highly complex tables have multi-layered nested structures, such as merged cells. These structures increase the difficulty of data extraction and understanding. Directly applying general strategies may lead to inaccurate results. Therefore, complex structure processing strategies are needed to process them to avoid information loss or incorrect parsing.

[0071] Step S33: Determine whether the remainder marker is positive.

[0072] If yes, proceed to step S34; otherwise, proceed to step S35.

[0073] In this embodiment, if the table complexity is low, it is determined whether the remainder marker is positive. If the remainder marker is positive, it indicates that there is data not explicitly listed in the table, and the missing information needs to be supplemented through knowledge reasoning. Directly applying the general strategy cannot process this type of table, resulting in incomplete information. At this time, the implicit entity expansion processing strategy is triggered, which uses an external knowledge base to perform set difference operations to derive the implicit entity list and inject the data to avoid information loss or incorrect parsing.

[0074] Step S34: Select the implicit entity augmentation processing strategy.

[0075] Step S35: Determine whether the global semantic entropy value is less than the first preset threshold.

[0076] If yes, proceed to step S36; otherwise, proceed to step S37.

[0077] In this embodiment, if the remainder is marked negative, it is determined whether the global semantic entropy value of the table is lower than a pre-set first preset threshold. If the global semantic entropy value is lower than the first preset threshold, it indicates that there is a large amount of duplicate information in the table data. Ignoring highly redundant data would waste storage space and reduce retrieval speed. Therefore, the processing flow of the redundant information compression strategy is triggered at this time to perform homogeneous clustering on highly redundant areas to reduce the amount of data processing.

[0078] Step S36: Select a redundant information compression processing strategy.

[0079] Step S37: Determine whether the blank rate is greater than the second preset threshold.

[0080] If yes, proceed to step S38; otherwise, proceed to step S39.

[0081] In this embodiment, if the global semantic entropy value is greater than a first preset threshold, it is determined whether the blank rate of the table exceeds a pre-set second preset threshold. If the blank rate is higher than the second preset threshold, it indicates that the table is interactive and requires the user to fill in information. Ignoring these interactive elements would affect the user experience of the table. Therefore, the processing flow of the interactive form filling template is triggered at this time, establishing a topological binding relationship between field labels and form filling slots based on spatial proximity, and identifying section titles with regional dominance characteristics.

[0082] Step S38: Select the interactive form filling template processing strategy.

[0083] Step S39: Select a general processing strategy.

[0084] In this embodiment, if the blank rate is less than or equal to the second preset threshold, it proves that the table is a simple table with low complexity, no redundant items, high global semantic entropy value and low blank rate. At this time, a general processing strategy can be selected to perform basic structured transformation and data extraction in order to efficiently complete the data processing task and avoid unnecessary resource consumption.

[0085] Please refer to Figure 4 ,for Figure 1 A flowchart illustrating the first practical manifestation of step S03 in a table processing method based on an adaptive strategy.

[0086] Based on the above embodiments, in some embodiments, when the processing strategy is a complex structured processing strategy, the processing of structured data based on the processing strategy mentioned in step S03 to obtain the corresponding metadata model may specifically include, for example: Figure 8 The steps shown are as follows: Step S41: Construct a virtual coordinate matrix and map the structured data into the virtual coordinate matrix.

[0087] To address the issues of information loss or erroneous parsing that can occur when processing complex structured tables, it is necessary to transform the originally irregular physical layout into a regular virtual coordinate system, thereby providing a unified foundation for subsequent data filling and semantic analysis.

[0088] In this embodiment, constructing a virtual coordinate matrix and mapping structured data into it means creating a virtual coordinate system corresponding to the physical layout of the table before processing the structured data. This virtual coordinate system consists of rows and columns, with each cell corresponding to a unique coordinate point. Subsequently, each cell in the structured data, regardless of whether it spans rows or columns, is mapped to a corresponding position in the virtual coordinate matrix.

[0089] In some embodiments, an M can be created based on the number of rows and columns of the table. A virtual coordinate matrix of N. Traverse the HTML structure tree, read the starting row and starting column coordinates of each cell, and map these coordinates to the corresponding positions in the virtual coordinate matrix.

[0090] Step S42: Copy the text value of the merged cell in the structured data and fill it into each virtual coordinate point in the logical placeholder field of the merged cell.

[0091] Because a cell spanning multiple rows and columns contains only one text value in the physical table, but occupies multiple coordinate points in the virtual coordinate matrix, the cell's content needs to be copied to all logical placeholder fields to ensure that each coordinate point contains complete information.

[0092] In this embodiment, the logical placeholder field refers to the area covered by the merged cells in the virtual coordinate matrix. By copying the content of the merged cells that span rows and columns to each virtual coordinate point they cover, each coordinate point contains the original content of the cell.

[0093] In some embodiments, a flood fill algorithm can be executed to copy and fill the text value of each merged cell that spans multiple virtual coordinate points to each virtual coordinate point within its logical placeholder. After processing, each virtual coordinate point in the virtual coordinate matrix is ​​assigned an independent atomic value, achieving physical de-overlapping and atomization of the data.

[0094] Step S43: For each virtual coordinate point in the virtual coordinate matrix, extract all the table headers of the virtual coordinate point, and concatenate all the table headers with the text value of the virtual coordinate point to obtain the full path semantic chain of the virtual coordinate point.

[0095] To provide sufficient contextual information so that the semantic understanding of the subsequent intelligent knowledge base is more accurate, it is necessary to link each cell with the header information to which it belongs, forming a complete full-path semantic chain.

[0096] Therefore, in this embodiment, for each virtual coordinate point in the virtual coordinate matrix, the row header, list header, and any nested hierarchical headers to which it belongs are obtained, and these header information are concatenated with the text value corresponding to the coordinate point to form a complete full-path semantic chain.

[0097] In some embodiments, for each virtual coordinate point, the header text can be extracted by traversing upwards along the row coordinates and the column text can be extracted by traversing backwards along the column coordinates. Then, these header information and the text value corresponding to the virtual coordinate point are concatenated in a specific order to form a full-path semantic chain.

[0098] Step S44: Fill the virtual coordinates of the virtual coordinate points and the full path semantic chain into the first metadata template to obtain the corresponding metadata model.

[0099] The first metadata template is the metadata template corresponding to the complex structured processing strategy.

[0100] In this embodiment, each virtual coordinate point and its corresponding full-path semantic chain are organized and encapsulated according to a predefined first metadata template to form a structured metadata model, so that the table processing results can be stored and subsequently applied in a standardized format.

[0101] In some embodiments, please refer to Figure 5 This is a schematic diagram of a complex structured table provided in an embodiment of the present invention. Figure 5 As shown, the table contains cells spanning multiple rows, such as "200", "Passenger Dedicated Line", "250", "Ballasted Track", "Ballastless Track", and "300", and the header level depth is greater than or equal to 2. Therefore, the table is considered to have high complexity. In this case, a complex structure processing strategy is selected to process the table, resulting in the following information: Control-related information: strategy tag (strategy_tag), with a fixed value of structural_restoration.

[0102] Spatial information: Virtual coordinate set (coordinate_set), single-point coordinates, formatted as [row, column]. Granularity type (granularity_type), fixed value is atomic level.

[0103] Semantic / Business Information: Natural Language Processing Description (nlp_description), Natural Language Semantic Chain, formatted as "{First-level header} - {Second-level header} - {Row attribute} has a value of {atomic value}", with a value of "Design specification context: Track engineering - Ballastless track - Minimum curve radius, under the design speed of 350km / h, its value is 7000 meters."

[0104] Populate this information into the first metadata template to obtain the corresponding metadata model: { "Unique Identifier": "Universal Unique Identifier_Type1_001", Semantic text: Design specification context: Track engineering - Ballastless track - Minimum curve radius, under design speed of 350km / h, is 7000 meters. "Metadata": { "Structural Information": { "Granularity": "Atomic level", "Positional Range": [[12, 4]]}, "Semantic information": { "Strategy label": "Structure recovery"}, "Business Information": { "Head Classification": "Minimum Curve Radius" "Line key": "Batterless track - 350km / h", Core value: 7000 } } Based on the above technical solution, this embodiment effectively solves the problem of traditional table processing methods struggling to extract complete information when dealing with merged cells by constructing a virtual coordinate matrix and performing data filling and semantic chain splicing. The resulting metadata model not only includes the physical location information of the table but also provides a complete semantic path, greatly improving the accuracy and understandability of subsequent knowledge retrieval.

[0105] Please refer to Figure 6 ,for Figure 1 A flowchart illustrating the second practical manifestation of step S03 in a table processing method based on an adaptive strategy.

[0106] Based on the above embodiments, in some embodiments, when the processing strategy is an implicit entity augmentation processing strategy, the processing of structured data based on the processing strategy mentioned in step S03 to obtain the corresponding metadata model may specifically include, for example: Figure 6 The steps shown are as follows: Step S51: Use the second regular expression to match the structured data, identify the target cells, and determine the header attributes of the column to which each target cell belongs.

[0107] The target cell contains more than one keyword.

[0108] In this embodiment, a predefined second regular expression is used to scan structured data to find cells containing the remaining keywords. Simultaneously, the header attributes of the columns containing these target cells are determined, such as "Applicable Region" or "Device Type".

[0109] Identifying the target cell is the starting point of the entire implicit entity expansion process, requiring accurate identification of information containing data that is not explicitly listed. Determining the column header attributes provides a crucial index for subsequent knowledge base queries, used to retrieve the relevant full set of entities from the pre-defined knowledge base.

[0110] Step S52: Construct an explicit entity set based on the text values ​​in the non-target cells.

[0111] In this embodiment, constructing an explicit entity set based on text values ​​in non-target cells means excluding target cells containing the remaining keywords from the table, extracting the text values ​​contained in the remaining cells, and organizing these text values ​​into a set, which is called an explicit entity set.

[0112] The explicit entity set represents all entities explicitly listed in the table and serves as the basis for comparison with the full entity set in the knowledge base. By comparing the explicit entity set and the full entity set, implicit entities not explicitly listed in the table can be identified.

[0113] Step S53: Obtain the full set of entities under the header attributes from the preset knowledge base, and determine the implicit entity set by the difference between the full set of entities and the displayed entity set.

[0114] In this embodiment, by utilizing the header attribute of the column containing the target cell, all possible entity sets corresponding to that header attribute are retrieved from a preset knowledge base. This complete entity set is then compared with the previously constructed explicit entity set using a difference operation. The result of this difference operation is the implicit entity set, representing entities not explicitly listed in the table.

[0115] A pre-defined knowledge base stores complete entity information within the domain, used to supplement missing information in the table. By calculating the difference between sets, entities not explicitly listed in the table can be accurately identified, thereby improving the knowledge base.

[0116] Step S54: Convert the implicit entity set into natural language description text, and inject the natural language description text as extended data into the data object of the target cell.

[0117] In this embodiment, the previously calculated implicit entity set is converted into natural language description text and added to the data object of the target cell as additional attributes or information to enrich the semantic information of the cell and improve the accuracy and comprehensiveness of subsequent retrieval.

[0118] For example, assuming the full entity set is "City A, City B, City C, City D, City E", and the explicit entity set is "City A, City B, City C", then the implicit entity set is "City D, City E". This implicit entity set can then be converted into natural language description text that "includes unlisted cities such as City E and City F", and injected as an extended semantic attribute into the data object of the target cell. This gives the previously vague "Other" cell a logically specific entity reference, enabling explicit matching during vector retrieval.

[0119] Step S55: Fill the physical coordinates and extended data of the target cell into the second metadata template to obtain the corresponding metadata model.

[0120] The second metadata template is the metadata template corresponding to the implicit entity augmentation processing strategy.

[0121] In this embodiment, the virtual coordinates and extended data of the target cell are organized and encapsulated according to a predefined second metadata template to form a structured metadata model. This facilitates the storage and subsequent application of the table processing results in a standardized format.

[0122] For example, please refer to Figure 7 This is a schematic diagram of an implicit entity extended table provided in an embodiment of the present invention. Figure 7 As shown, the table contains the description "Other Districts," indicating that the remainder is marked positive. This suggests that the table contains data that is not explicitly listed, meaning that City C has other districts besides C1, C2, C3, and C4. This triggers the implicit entity expansion processing strategy, using a pre-defined knowledge base to perform set difference operations and derive an implicit entity list (e.g., City C also includes C5, C6, and C7). This list is then injected as extended data into the target cell's data object, resulting in the following information: Control-related information: The strategy tag (strategy_tag) has a fixed value of entity_expansion, which activates the semantic information expansion field (semantic_info.expansion) in the JSON schema and records the expansion history.

[0123] Spatial information: The coordinate set (coordinate_set) is the coordinates of a single point, in the format [row, column]; the granularity type (granularity_type) is fixed at the atomic level.

[0124] Semantic / Business Information: The Natural Language Description field (nlp_description) is semantically enhanced text, with an example format of "{header attribute} is {original cell value}. Specifically, it includes the following implicit entities: {first N items of the implicit entity list}". The business context (business_context) includes column attributes (header_category), row primary key (row_key), and original cell text (core_value).

[0125] Extended data (expansion_data): Contains complete implicit and explicit information, which is used for subsequent data auditing or front-end "floating tooltip" display.

[0126] Fill this information into the second metadata template to obtain the corresponding metadata model: { "Unique Identifier": "Universal Unique Identifier_Type2_002", Semantic Text: "Applicable area: other regions (i.e., excluding City A and City B). Includes implicit entities: C5, C6, C7, etc. Accommodation fees are 800, 450, and 310 respectively." "Metadata": { "Structural Information": { "Granularity": "Atomic level", "Location Range": [[8, 2]]}, Semantic information: { "Strategy Tag": "Entity Expansion" Extended Data: ["C5 Zone", "C6 Zone", "C7 Zone"] }, "Business Information": { "Header Category": "Applicable Region" "Core Values": "Other Areas" } } } Based on the above technical solution, the implicit entity expansion processing method of this embodiment effectively solves the problem of information gaps in existing knowledge bases, improving the comprehensiveness and accuracy of retrieval. By injecting extended data into the data object of the target cell and organizing it with a second metadata template, the content of the knowledge base is enriched, the value of data utilization is enhanced, and retrieval failures or errors caused by information gaps are avoided.

[0127] Please refer to Figure 8 ,for Figure 1 A flowchart illustrating the third practical manifestation of step S03 in a table processing method based on an adaptive strategy.

[0128] Based on the above embodiments, in some embodiments, when the processing strategy is a redundant information compression processing strategy, the processing of structured data based on the processing strategy mentioned in step S03 to obtain the corresponding metadata model may specifically include, for example: Figure 8 The steps shown are as follows: Step S61: Cluster the data regions in the structured data to obtain multiple data clusters.

[0129] In this embodiment, by clustering the cell content within a data region of structured data, cells with similar characteristics are grouped together to form multiple data clusters. A data cluster is a set of cells with common features. Through clustering, repetitive information can be condensed into a few data clusters, thereby reducing redundant information and facilitating subsequent summary generation.

[0130] In some embodiments, based on the principle of semantic consistency, cells with the same content and that are physically consecutive (or belong to the same category in logical columns) can be grouped into the same data cluster. For each data cluster, the set of coordinates it covers and the range of rows and columns are calculated, thereby converting P independent cells into Q aggregated data blocks (where Q is much smaller than P).

[0131] Step S62: Perform core feature extraction processing on each data cluster to obtain the core business value of each data cluster.

[0132] To concisely summarize the content of each data cluster, it is necessary to extract the core features or values ​​that represent the cluster for subsequent summary text generation. In this embodiment, by analyzing all cells in each data cluster, common features or values ​​that represent the cluster are extracted and defined as core business values. Core business values ​​are a high-level summary of the data content of the data cluster.

[0133] In some embodiments, a unique common attribute value (i.e., the text content shared by all cells within the data cluster) can be extracted from each data cluster as the core business value of that data cluster. Simultaneously, the set of row header primary keys or column header attributes covered by the data cluster is extracted as business boundary conditions defining the scope of effectiveness of the core value.

[0134] Step S63: Generate the corresponding summary text based on the core business values.

[0135] In this embodiment, a concise natural language description is generated based on the core business values ​​of each data cluster to summarize the data content contained in that cluster. The summary text is a compressed representation of the original data, which can express the content of the data cluster in a more concise and understandable way, reducing storage space and facilitating user comprehension.

[0136] In some embodiments, a summary text describing the data cluster can be generated based on a "range + value" template. For example, "Row 1 is A, Row 2 is A...Row 10 is A" can be compressed into a natural language statement "The state of rows 1 to 10 is A", thereby achieving semantic dimensionality reduction of low-entropy information.

[0137] Step S64: Fill the third metadata template with all physical coordinates and summary text covered by each data cluster to obtain the corresponding metadata model.

[0138] The third metadata template is the metadata template corresponding to the redundant information compression processing strategy.

[0139] In this embodiment, the coordinate information covered by each data cluster and the generated summary text are organized and encapsulated according to a predefined third metadata template to form a structured metadata model, so as to realize the storage and subsequent application of the table processing results in a standardized format.

[0140] For example, please refer to Figure 9 This is a schematic diagram of a redundant information compressed table provided in an embodiment of the present invention. Figure 9 As shown, the table contains a large number of cells with the text value "manual / parking", indicating a large amount of duplicate information. Since the global semantic entropy value is less than the first preset threshold, the redundant information compression processing strategy is triggered. Homogeneous clustering is performed on the highly redundant areas, resulting in the following information: Control-related information: The strategy tag has a fixed value of entropy reduction.

[0141] Spatial information: Coordinate set (coordinate_set), which represents all physical coordinate points covered by the data cluster, in the format "[row 1, column 1], [row 2, column 2], ..." or "[start row, column], [end row, column]". Granularity type (granularity_type) is fixed at the group level.

[0142] Semantic / Business Information: The Natural Language Description field (nlp_description) is a rule summary, with a structure format such as "For {business boundary set}, its {column attributes} are all {core business values}". The business context (business_context) includes column attributes (header_category), the list of aggregated primary keys (row_key), and public values ​​(core_value).

[0143] Compression statistics include the number of original cells, the number of clustered objects, and the compression ratio, which are used to evaluate the system's processing performance.

[0144] Populate this information into the third metadata template to obtain the corresponding metadata model: { "Unique Identifier": "Universal Unique Identifier_Type3_003", Semantic Text: "Rule Summary: The track circuit equipment within the K10+200 to K12+500 section is operating normally." "Metadata": { "Structure Information": { "Granularity": "Group level" "Location range": [[10, 3], [25, 3]] }, Semantic information: { Strategy tag: "Entropy reduction" "Compression Statistics": { "Compression Ratio": "16:1", "Data Volume": 16} }, "Business Information": { "Header Category": "Running Status" Core value: "Normal" } } } Based on the above technical solution, the redundant information compression processing method in this embodiment can alleviate the problem of tabular data redundancy and improve storage efficiency and retrieval performance. This compression strategy not only reduces the storage space occupied by the knowledge base, but also improves the retrieval speed and avoids scanning a large amount of duplicate data, thereby improving the overall system efficiency.

[0145] Please refer to Figure 10 ,for Figure 1 A flowchart illustrating the fourth practical manifestation of step S03 in a table processing method based on an adaptive strategy.

[0146] Based on the above embodiments, in some embodiments, when the processing strategy is an interactive form-filling template-based processing strategy, the processing of structured data based on the processing strategy mentioned in step S03 to obtain the corresponding metadata model may specifically include, for example: Figure 10 The steps shown are as follows: Step S71: Based on the content characteristics of each cell in the structured data, divide the cell into labels or slots.

[0147] Interactive forms need to clearly distinguish which cells are labels for describing fields and which cells are slots for users to fill in data in order to build a clear form structure and provide effective user guidance.

[0148] In this embodiment, each cell in the structured data is analyzed and categorized into "labels" or "slots" based on its content. Labels describe the meaning of the field, while slots indicate the location where the user needs to fill in the information.

[0149] In some embodiments, non-empty text cells can be identified as field labels and blank cells or cells containing only placeholders can be identified as data entry slots based on preset rules and feature libraries.

[0150] Step S72: Establish a mapping relationship between the slot and the nearest label to form a label-slot pair.

[0151] In this embodiment, by associating the identified slots with the tags that are adjacent to them in physical location, a tag-slot pair is formed, so that the intelligent knowledge base can understand the correspondence between tags and slots, ensuring that users can accurately understand the meaning of the field corresponding to each slot when querying.

[0152] In some embodiments, based on the principle of spatial proximity, each slot can be mapped to the label that is closest to it in physical distance, forming a label-slot pair.

[0153] Step S73: Perform semantic parsing on the tag-slot pairs to extract the filling constraints.

[0154] In this embodiment, by analyzing the information of labels and slots, relevant data entry constraints are extracted, such as required fields, data formats, and length limits. These constraints guide users to fill out the form correctly, ensuring the accuracy and completeness of the data.

[0155] In some embodiments, text analysis can be performed on the content of tags and slots to identify preset interactive prompts, and then the filling constraints can be extracted and recorded based on the prompts.

[0156] Furthermore, cells with region dominance characteristics (i.e., text cells spanning more than a preset number of columns or rows) can be identified as section headings. Then, based on geometric inclusion relationships, all tag-slot pairs within the jurisdiction of that heading are clustered to generate logically independent data entry groups, thereby further improving the understanding capabilities of the intelligent knowledge base.

[0157] Step S74: Fill the physical coordinates and reporting constraints of the tag-slot pair into the fourth metadata template to obtain the corresponding metadata model.

[0158] Among them, the fourth metadata template is the metadata template corresponding to the interactive data entry template-type processing strategy.

[0159] In this embodiment, the coordinate information of the tag-slot pair and the filling constraints are organized and encapsulated according to the predefined fourth metadata template to form a structured metadata model, so as to realize the storage and subsequent application of the table processing results in a standardized format.

[0160] For example, please refer to Figure 11 This is a schematic diagram of an interactive form-filling template provided in an embodiment of the present invention. Figure 11 As shown, the table contains a large number of blank cells, with a blank rate exceeding the second preset threshold, indicating that the table is an interactive form template. In this case, the processing flow of the interactive form template handling strategy is triggered, resulting in the following information: Natural language semantic string: such as "Basic information area. Includes name (required), phone number (required), and remarks. 11-digit mobile phone number is required".

[0161] Structured interaction list: such as "{"field name": "name", "required": true}, {"field name": "remarks", "required": false}.

[0162] Populate this information into the fourth metadata template to obtain the corresponding metadata model: { "Unique Identifier": "Universal Unique Identifier_Type4_004", "Semantic Text": "Filling Guidelines: [Responsible Person Information] area. Includes required fields: Responsible Person (must sign), Contact Number." "Metadata": { "Structural Information": { "Granularity": "Group Level", "Location Range": [[5, 1], [8, 4]]}, "Semantic Information": { "Strategy Label": "Interactive Extraction"}, "Interactive Information": { "Region Title": "Responsible Person Information", "Field List": [ { "Field Name": "Operation Supervisor", "Required": true, "Identifier": " }, { "Field Name": "Contact Number", "Required": true, "Identifier": "(Required)"} ] } } } Based on the above technical solution, the interactive form-filling template processing method in this embodiment can extract structured information from interactive forms and provide user guidance. It can transform unstructured interactive forms into metadata models with clear structures and rules, thereby improving user experience and data utilization efficiency.

[0163] Please refer to Figure 12 The flowchart below shows another table processing method based on an adaptive strategy provided by an embodiment of the present invention, which includes the following steps: Step S81: Obtain tabular data and generate structured data based on the tabular data.

[0164] The specific implementation process of this step is the same as that of step S01, and will not be repeated here.

[0165] Step S82: Determine the characteristics of the structured data and select the corresponding processing strategy based on the characteristics.

[0166] The specific implementation process of this step is the same as that of step S02, and will not be repeated here.

[0167] Step S83: Process the structured data based on the processing strategy to obtain the corresponding metadata model.

[0168] The metadata model includes natural language description fields.

[0169] The specific implementation process of this step is the same as that of step S03, and will not be repeated here.

[0170] Step S84: Convert the natural language description field into vector features.

[0171] In this embodiment, converting the natural language description field into a vector feature means transforming the natural language description (i.e., the "semantic_text" field) in the metadata model into a fixed-dimensional vector using a pre-trained dedicated embedding model. This vector can represent the semantic information of the natural language description.

[0172] Since traditional keyword-based retrieval cannot capture the deep semantic information of text, while vector representation can more accurately reflect the meaning of text, this embodiment converts text information into numerical vectors to calculate semantic similarity in order to achieve semantic similarity-based retrieval.

[0173] Step S85: Construct index entries in the preset vector database.

[0174] The index entries include a vector index area and a scalar load area.

[0175] In this embodiment, constructing an index entry in a preset vector database means creating an index entry for each metadata model in the vector database. This index entry contains two areas: a vector index area for storing vector features and a scalar payload area for storing the serialized representation of the original data.

[0176] Step S86: Serialize the metadata model and store the processing result in the scalar payload area of ​​the index entry.

[0177] In this embodiment, serializing the metadata model and storing the processing result in the scalar payload area of ​​the index entry means using a serialization algorithm (such as JSON serialization) to convert the JSON structure of the metadata model into a format that can be stored in the scalar payload area, such as string or binary data, and storing the converted data in the scalar payload area of ​​the index entry to facilitate the retrieval of the original data.

[0178] Step S87: Store the vector features corresponding to the metadata model into the vector index area of ​​the index entry. In this embodiment, storing the vector features corresponding to the metadata model into the vector index area of ​​the index entry means storing the previously generated vector features into the vector index area of ​​the index entry so that the vector database can perform efficient similarity search.

[0179] Based on the above technical solution, this embodiment transforms natural language description fields into vector features, enabling semantic similarity-based retrieval and avoiding the limitations of traditional keyword-based retrieval. This vectorized representation can capture the deep semantic information of the text, allowing the system to better understand user intent and return more relevant results.

[0180] Please refer to Figure 13 This is a schematic diagram of the structure of a table processing device based on an adaptive strategy provided in an embodiment of the present invention. Figure 13 As shown, the table processing device based on the adaptive strategy may include: The preprocessing module 100 is used to acquire tabular data and generate structured data based on the tabular data; The feature extraction and decision module 200 is used to determine the features of structured data and select the corresponding processing strategy based on the features; The data processing module 300 is used to process structured data based on processing strategies to obtain the corresponding metadata model.

[0181] Please refer to Figure 14 This is a schematic diagram of another table processing device based on an adaptive strategy provided in an embodiment of the present invention. Figure 14 As shown: Based on the above embodiments, in some embodiments, the preprocessing module 100 may include: The visual layout analysis submodule 110 is used to perform primitive detection on the input document, identify table borders, background color blocks and dividing lines, strip away non-table areas such as headers and footers, and determine the physical boundaries of the table. The structured transformation submodule 120 is used to extract the text content in the table and transform the visual table into structured data containing row, column, spanning row, and spanning column attributes.

[0182] Based on the above embodiments, in some embodiments, the feature extraction and decision module 200 may include a multi-dimensional feature calculation submodule 210, which is used for: The complexity of a table is determined based on the span attribute of the structured data; The first regular expression is used to match structured data, and the interactive markers of the table are determined based on the first matching result; the first regular expression is used to match preset interactive keywords. The second regular expression is used to match structured data, and the remaining item tags of the table are determined based on the second matching result; the second regular expression is used to match preset remaining item keywords; The global semantic entropy value of a table is determined based on the frequency of text repetition in structured data. The blank rate of a table is determined based on the number of blank cells in the structured data; Complexity, interaction markers, remainder markers, global semantic entropy, and blank rate are identified as features of structured data.

[0183] Based on the above embodiments, in some embodiments, the feature extraction and decision module 200 may include a decision routing submodule 220, which is used for: Determine if the complexity is high; If so, then choose the complex structure processing strategy; If not, then check if the remainder flag is positive; If the remaining item is marked as positive, then the implicit entity augmentation processing strategy is selected; If the remaining item is marked as negative, then determine whether the global semantic entropy value is less than the first preset threshold. If the global semantic entropy value is less than the first preset threshold, then a redundant information compression processing strategy is selected. If the global semantic entropy value is greater than or equal to the first preset threshold, then determine whether the blank rate is greater than the second preset threshold. If the blank rate is greater than the second preset threshold, then the interactive form filling template processing strategy will be selected. If the blank rate is less than or equal to the second preset threshold, then the general processing strategy is selected.

[0184] Based on the above embodiments, in some embodiments, the data processing module 300 may include a complex case structure table processing submodule 310, which includes: Virtual coordinate mapping unit 311, used for: Construct a virtual coordinate matrix and map the structured data into the virtual coordinate matrix; Copy the text value of the merged cell in the structured data and fill it into each virtual coordinate point within the logical placeholder area of ​​the merged cell; The semantic path concatenation unit 312 is used to extract all the table headers of each virtual coordinate point in the virtual coordinate matrix, and concatenate all the table headers with the text value of the virtual coordinate point to obtain the full path semantic chain of the virtual coordinate point. The first encapsulation unit 313 is used to fill the virtual coordinates of the virtual coordinate points and the full path semantic chain into the first metadata template to obtain the corresponding metadata model; the first metadata template is the metadata template corresponding to the complex structure processing strategy.

[0185] Based on the above embodiments, in some embodiments, the data processing module 300 may include an implicit entity extended table processing submodule 320, which includes: Set difference operation unit 321 is used for: The second regular expression is used to match structured data, identify target cells, and determine the header attributes of the column to which each target cell belongs; the target cells contain additional keywords; Construct an explicit entity collection based on the text values ​​in non-target cells; Obtain the full set of entities under the header attributes from the preset knowledge base, and determine the implicit entity set by the difference between the full set of entities and the displayed entity set. Semantic injection unit 322 is used to convert the implicit entity set into natural language description text and inject the natural language description text as extended data into the data object of the target cell; The second encapsulation unit 323 is used to fill the physical coordinates and extended data of the target cell into the second metadata template to obtain the corresponding metadata model; the second metadata template is the metadata template corresponding to the implicit entity expansion processing strategy.

[0186] Based on the above embodiments, in some embodiments, the data processing module 300 may include a redundant information compression table processing submodule 330, which includes: Data clustering unit 331 is used for: Clustering is performed on data regions in structured data to obtain multiple data clusters; Core features are extracted from each data cluster to obtain the core business value for each data cluster; Summary generation unit 332 is used to generate corresponding summary text based on core business values; The third encapsulation unit 333 is used to fill all physical coordinates and summary text covered by each data cluster into the third metadata template to obtain the corresponding metadata model; the third metadata template is the metadata template corresponding to the redundant information compression processing strategy.

[0187] Based on the above embodiments, in some embodiments, the data processing module 300 may include an interactive form filling template processing submodule 340, which includes: The table element recognition unit 341 is used to divide the cell into labels or slots based on the content features of each cell in the structured data; The visual anchor point detection unit 342 is used for: Establish a mapping relationship between the slot and the nearest label to form a label-slot pair; Perform semantic parsing on the tag-slot pairs to extract the filling constraints; The fourth encapsulation unit 343 is used to fill the physical coordinates and filling constraints of the tag-slot pair into the fourth metadata template to obtain the corresponding metadata model; the fourth metadata template is the metadata template corresponding to the interactive filling template type processing strategy.

[0188] Based on the above embodiments, in some embodiments, the metadata model includes natural language description fields; The table processing device based on the adaptive strategy may further include a vector adaptation and storage module 400, which includes: Vector encoding submodule 410 is used to convert natural language description fields into vector features; Index writing submodule 420 is used for: Construct index entries in the pre-defined vector database; each index entry includes a vector index area and a scalar payload area. The metadata model is serialized, and the processing result is stored in the scalar payload area of ​​the index entry. Store the vector features corresponding to the metadata model into the vector index area of ​​the index entry.

[0189] This embodiment provides an electronic device, including a processor and a memory. The memory is used to store at least one instruction. When the instruction is loaded and executed by the processor, it implements the table processing method based on the adaptive strategy described above. Its execution method and beneficial effects are similar and will not be repeated here.

[0190] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A table processing method based on an adaptive strategy, characterized in that, include: Obtain tabular data and generate structured data based on the tabular data; Determine the characteristics of the structured data, and select the corresponding processing strategy based on the characteristics; The structured data is processed based on the aforementioned processing strategy to obtain the corresponding metadata model.

2. The method according to claim 1, characterized in that, Determining the characteristics of the structured data includes: The complexity of the table is determined based on the span attribute of the structured data; The structured data is matched using a first regular expression, and the interactive markers of the table are determined based on the first matching result; the first regular expression is used to match preset interactive keywords. The structured data is matched using a second regular expression, and the remaining item markers of the table are determined based on the second matching result; the second regular expression is used to match preset remaining item keywords. The global semantic entropy value of the table is determined based on the text repetition frequency of the structured data; The blank rate of the table is determined based on the number of blank cells in the structured data. The complexity, the interaction marker, the remainder marker, the global semantic entropy value, and the blank rate are determined as features of the structured data.

3. The method according to claim 2, characterized in that, The processing strategy based on the feature selection includes: Determine whether the complexity is high. If so, then choose the complex structure processing strategy; If not, then determine whether the remaining item flag is positive; If the remaining item is marked as positive, then the implicit entity augmentation processing strategy is selected; If the remainder is marked as negative, then determine whether the global semantic entropy value is less than a first preset threshold. If the global semantic entropy value is less than the first preset threshold, then a redundant information compression processing strategy is selected. If the global semantic entropy value is greater than or equal to the first preset threshold, then determine whether the blank rate is greater than the second preset threshold; If the blank rate is greater than the second preset threshold, then the interactive form filling template processing strategy is selected; If the blank rate is less than or equal to the second preset threshold, then a general processing strategy is selected.

4. The method according to claim 3, characterized in that, When the processing strategy is the complex structured processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Construct a virtual coordinate matrix and map the structured data to the virtual coordinate matrix; Copy the text value of the merged cell in the structured data and fill it into each virtual coordinate point within the logical placeholder field of the merged cell; For each virtual coordinate point in the virtual coordinate matrix, extract all the headers of the virtual coordinate point, and concatenate all the headers with the text value of the virtual coordinate point to obtain the full path semantic chain of the virtual coordinate point; The virtual coordinates of the virtual coordinate points and the full path semantic chain are filled into the first metadata template to obtain the corresponding metadata model; the first metadata template is the metadata template corresponding to the complex structure processing strategy.

5. The method according to claim 3, characterized in that, When the processing strategy is the implicit entity augmentation processing strategy, the processing of the structured data based on the processing strategy to obtain the corresponding metadata model includes: The structured data is matched using the second regular expression to identify target cells and determine the header attribute of the column to which each target cell belongs; the target cell contains the remaining keywords; Construct an explicit entity collection based on the text values ​​in non-target cells; Obtain the full set of entities under the header attribute from the preset knowledge base, and determine the implicit entity set by the difference between the full set of entities and the displayed entity set. The implicit entity set is converted into natural language description text, and the natural language description text is injected as extended data into the data object of the target cell; The physical coordinates of the target cell and the extended data are filled into the second metadata template to obtain the corresponding metadata model; the second metadata template is the metadata template corresponding to the implicit entity expansion processing strategy.

6. The method according to claim 3, characterized in that, When the processing strategy is the redundant information compression processing strategy, the step of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Clustering is performed on the data regions in the structured data to obtain multiple data clusters; Perform core feature extraction processing on each data cluster to obtain the core business value of each data cluster; Generate corresponding summary text based on the core business values; The physical coordinates covered by each data cluster and the summary text are filled into the third metadata template to obtain the corresponding metadata model; the third metadata template is the metadata template corresponding to the redundant information compression processing strategy.

7. The method according to claim 3, characterized in that, When the processing strategy is the interactive form-filling template-based processing strategy, the process of processing the structured data based on the processing strategy to obtain the corresponding metadata model includes: Based on the content characteristics of each cell in the structured data, the cells are divided into labels or slots; Establish a mapping relationship between the slot and the nearest tag to form a tag-slot pair; Semantic parsing is performed on the tag-slot pairs to extract the filling constraints; The physical coordinates of the tag-slot pair and the filling constraints are filled into the fourth metadata template to obtain the corresponding metadata model; the fourth metadata template is the metadata template corresponding to the interactive filling template type processing strategy.

8. The method according to any one of claims 1-7, characterized in that, The metadata model includes natural language description fields; After processing the structured data based on the processing strategy to obtain the corresponding metadata model, the method further includes: The natural language description fields are converted into vector features; An index entry is constructed in a pre-defined vector database; the index entry includes a vector index area and a scalar payload area; The metadata model is serialized, and the processing result is stored in the scalar payload area of ​​the index entry. The vector features corresponding to the metadata model are stored in the vector index area of ​​the index entry.

9. A table processing system based on an adaptive strategy, characterized in that, include: A preprocessing module is used to acquire tabular data and generate structured data based on the tabular data; The feature extraction and decision-making module is used to determine the features of the structured data and select the corresponding processing strategy based on the features; The data processing module is used to process the structured data based on the processing strategy to obtain the corresponding metadata model.

10. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store at least one instruction, which, when loaded and executed by the processor, implements the table processing method based on an adaptive strategy as described in any one of claims 1-8.

Images