Scientific and technical literature table knowledge chain generation method based on large language model

Abstract

The invention provides a scientific and technical literature table knowledge chain generation method based on a large language model, and relates to the technical field of semantic processing, and the method comprises the steps: receiving a table image, a PDF extraction region and a structured source code from scientific and technical literature, and generating initial input information in combination with table context information; calling a multi-modal large language model to carry out table content identification and structured processing; performing iterative self-correction based on the initial HTML structure representation and the original table image and table context information; based on a preset scientific entity ontology, semantic information extraction is carried out on the high-fidelity HTML structure representation; and organizing a knowledge chain according to a preset logic normal form and the entity extraction result. According to the method and the device, the technical problem that in the prior art, identification errors are likely to occur when complex and non-standardized tables are processed, and the content identification precision is insufficient due to insufficient deep logic relation mining capacity can be solved, and the precision of table content identification and structure analysis is improved.

Description

Technical Field

[0001] This invention relates to the field of semantic processing technology, specifically to a method for generating knowledge chains in scientific and technological literature tables based on a large language model. Background Technology

[0002] Scientific literature serves as a crucial vehicle for knowledge accumulation and technological innovation. Especially in scientific research, tables, as a highly structured and information-dense form of expression, carry a wealth of experimental data, process parameters, and research conclusions. Existing table content recognition methods, including rule-based, machine learning, deep learning, and multimodal large language models, still have many limitations when dealing with complex and varied tables. Particularly for tables containing multi-level headers, merged cells, special symbols, and mixed text and graphics, the accuracy of structural analysis and text extraction still has significant room for improvement, easily introducing structural biases and content recognition fallacies, leading to a decline in semantic understanding. Furthermore, the diverse formats of table data in scientific literature and the lack of unified standards for semantic expression mean that existing methods often focus on extracting explicit data points while neglecting the complex scientific logic and deep-seated implicit dependencies hidden within tables, limiting the in-depth application of knowledge and resulting in insufficient content recognition accuracy.

[0003] In summary, existing technologies suffer from technical problems such as the susceptibility to recognition errors when processing complex and non-standardized tables, insufficient ability to mine deep logical relationships, resulting in insufficient content recognition accuracy and thus affecting the accuracy of knowledge extraction. Summary of the Invention

[0004] This application provides a method for generating knowledge chains in scientific and technological literature tables based on a large language model. This method addresses the technical problems in existing technologies, such as the ease with which recognition errors occur when processing complex and non-standardized tables, and the insufficient ability to mine deep logical relationships, which leads to insufficient content recognition accuracy and thus affects the accuracy of knowledge extraction.

[0005] In view of the above problems, this application provides a method for generating knowledge chains of scientific and technological literature tables based on a large language model. The method includes: receiving a table image, PDF extracted regions, and structured source code from a scientific and technological literature document; generating initial input information by combining table context information; calling a multimodal large language model to perform table content recognition and structuring processing on the initial input information to generate an initial HTML structure representation, including table row and column information, cell content, and physical layout information; performing iterative self-correction based on the initial HTML structure representation, the original table image, and the table context information to output a high-fidelity HTML structure representation; extracting semantic information from the high-fidelity HTML structure representation based on a preset scientific entity ontology to output entity extraction results; and organizing the extracted entities into a knowledge chain according to a preset logical paradigm and the entity extraction results.

[0006] Optionally, for the content of each cell in the initial HTML structure representation, semantic alignment processing of cell content is performed by combining logical position, original table image and table context information to output a first corrected structured table; cross-cell content coherence and logical consistency correction is performed on the first corrected structured table to generate a second corrected structured table; physical structure and layout correction is performed on the second corrected structured table to generate a third corrected structured table; semantic fidelity final check is performed on the third corrected structured table to output the high-fidelity HTML structure representation.

[0007] Optionally, the semantic alignment of cell content includes the correction of character errors, improper word segmentation, and misrecognition of numbers and special symbols.

[0008] Optionally, the cross-cell content in the first corrected structured table is extracted, specifically including content organization across rows and columns; it is determined whether the cross-cell content has one or more of the following: logical break, misalignment, or semantic conflict, and the problematic cell content is generated; the problematic cell content is corrected and adjusted to generate the second corrected structured table.

[0009] Optionally, the table physical structure and layout correction includes matching the number of rows and columns, as well as the correct identification and representation of merged cells.

[0010] Optionally, the semantic fidelity final check includes consistency correction between the content, structure, and the overall semantic level expressed by the combination of content and structure and the original table information.

[0011] Optionally, the table context information includes the table's official title, a unique number or citation identifier in the literature, and narrative text paragraphs surrounding the table.

[0012] Optionally, based on the preset scientific entity ontology, a first prompt message is constructed to guide the large language model to perform entity recognition; the first prompt message is input into the large language model to drive the large language model to perform entity recognition tasks, perform deep semantic analysis on the high-fidelity HTML structure representation, and generate the entity extraction result.

[0013] Optionally, the complete definition and detailed rules of the preset scientific entity ontology are used to clarify the scope and standards of entity recognition; the task instructions are used to guide the large language model to perform entity recognition tasks in structured table content and output the text span, category and related meta-information of each entity instance; the structured table content to be processed; the context information related to the high-fidelity HTML structure, used to supplement semantic understanding; and at least one labeled entity extraction example, used as an example to guide the large language model to understand the task requirements and output format.

[0014] Optionally, the preset logical paradigm is a condition-result knowledge chain model; a second prompt message is constructed to prompt the generation of knowledge chains; based on the second prompt message, logical reasoning and semantic analysis are performed on the entity extraction results to automatically identify and combine entities belonging to conditions and results, and construct multiple knowledge chains that meet the structural requirements, outputting a JSON list of knowledge chains containing multiple elements.

[0015] The technical solution provided in this application has at least the following beneficial effects:

[0016] By receiving table images, extracted PDF regions, and structured source code from scientific literature, and combining them with table context information, initial input information is generated. A multimodal large language model is then invoked to perform table content recognition and structuring processing on the initial input information, generating an initial HTML structure representation, including table row and column information, cell content, and physical layout information. Iterative self-correction is performed based on the initial HTML structure representation, the original table image, and the table context information, outputting a high-fidelity HTML structure representation. Semantic information is extracted from the high-fidelity HTML structure representation based on a preset scientific entity ontology, outputting entity extraction results. Finally, according to a preset logical paradigm, the extracted entities are organized into a knowledge chain based on the entity extraction results. In other words, by receiving table data from scientific literature and integrating relevant context information, invoking a multimodal large language model for semantic processing to generate an initial HTML structure representation, iteratively self-correcting the initial HTML structure representation against the original table image and table context information to correct errors in the initial recognition process, performing semantic extraction based on a preset scientific entity ontology, and organizing a knowledge chain based on a preset logical paradigm, the accuracy of table content recognition and structure parsing is improved, thereby enhancing the accuracy of semantic recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the method for generating knowledge chains in scientific and technological literature tables based on a large language model, as described in this application.

[0019] Figure 2 This is a flowchart illustrating the process of outputting a high-fidelity HTML structure representation in the scientific and technological literature table knowledge chain generation method based on a large language model, as described in this application. Detailed Implementation

[0020] This application provides a method for generating knowledge chains from scientific literature tables based on a large language model. This addresses the technical problems in existing technologies where recognition errors easily occur when processing complex and non-standardized tables, and the method suffers from insufficient deep logical relationship mining capabilities, leading to inaccurate content recognition and consequently affecting the accuracy of knowledge extraction. By receiving scientific literature table data and integrating relevant contextual information, a multimodal large language model is invoked for semantic processing to generate an initial HTML structure representation. The initial HTML structure representation is then iteratively self-corrected against the original table image and table context information to correct errors in the initial recognition process. Semantic extraction is performed by combining a preset scientific entity ontology, and the knowledge chain is organized using a preset logical paradigm. This improves the accuracy of table content recognition and structure parsing, thereby enhancing the accuracy of semantic recognition.

[0021] Examples, such as Figure 1 As shown, this application provides a method for generating knowledge chains for scientific and technological literature tables based on a large language model. Specifically, this method includes the following steps:

[0022] It receives table images, PDF extracted regions, and structured source code from scientific and technological literature, and generates initial input information by combining the table context information.

[0023] Specifically, it receives table images (such as scanned copies or screenshots), extracted PDF regions, and structured source code from various scientific and technological documents through a standardized input interface. The structured source code includes preliminary structured table data (such as HTML or XML fragments) parsed from document editing languages ​​like LaTeX. For example, extracting the PDF region corresponding to a table from a paper's PDF version (resulting in a JSON file containing similar content in table text blocks format). By providing a standardized data input interface, it receives user-submitted or automatically retrieved scientific and technological documents from a database. Supported input formats include, but are not limited to, raster image files of tables (such as PNG, JPEG, TIFF, etc.), table region images extracted from PDF documents using tools, or preliminary identified text / structured data, and table markup code contained in document source files such as LaTeX (e.g., paper.tex), WordML, and HTML. HTML is Hypertext Markup Language, an internationally standardized markup language widely used for creating and organizing the content structure of web pages and web applications. HTML was chosen as the preferred format for structured table representations because it allows for hierarchical tagging (such as...). 、 、 It has the ability to accurately describe the physical layout of a table (including row and column relationships, cell merging, header hierarchy, etc.) and supports embedding metadata.

[0024] Simultaneously, it possesses the ability to acquire and effectively utilize contextual information closely related to the target table, including the table's official caption, its unique number or citation identifier in the literature, and the narrative text paragraphs surrounding the table. The table caption is usually placed directly above or below the table, briefly summarizing its content. Table citation information is the table's official number in the literature (e.g., ...). Figure 1 (Table 2, etc.) and possible aliases. Adjacent text paragraphs are the text paragraphs preceding and following the table's position in the original text, containing important supplementary explanations of the table's data background, experimental methods, key findings, or data interpretation. All input data and contextual information will be uniformly formatted and passed to subsequent processing modules.

[0025] All received raw table data (whether images, text, or structured source code) and captured contextual metadata (titles, numbers, adjacent text) are not directly passed to subsequent modules. Instead, they undergo a unified formatting preprocessing stage to convert data and contextual information from different languages ​​(image pixels, PDF text blocks, LaTeX code) into an internally defined, unified intermediate data structure (such as a composite object containing image matrices, text strings, structured tags, and associated contextual fields).

[0026] For example, consider a table image from a scientific document. The table displays the experimental results of the thermal stability of a certain material. The table in the image contains multi-level headers and merged cells. The table content was extracted using OCR (Optical Character Recognition) technology. Simultaneously, contextual information was extracted from the document: Table Caption: The experimental results are an assessment of the thermal stability of different materials; Table Citation Information: The table number is Table 2; Adjacent Text Paragraph: In this experiment, the thermal stability of three different materials was tested under the conditions of 200°C for 4 hours, and the results showed… (details omitted).

[0027] By accepting tabular data from different sources and flexibly adapting to various input formats (such as images, PDFs, and structured data), the ability to process complex tables is enhanced. By binding tabular data with its closely related contextual information (such as headings, citations, and surrounding text), it helps to understand the precise meaning of the table and eliminate ambiguity.

[0028] The initial input information is processed by calling a multimodal large language model to perform table content recognition and structuring, generating an initial HTML structure representation, which includes table row and column information, cell content and physical layout information.

[0029] Specifically, the Multimodal Large Language Model (MLLM) serves as the core engine, responsible for initially transforming the input tabular data (especially image format) into a machine-readable, universally applicable structured representation format, preferably HTML. Pre-trained MLLM models with excellent performance in both visual understanding and text generation are selected, including but not limited to the Qwen-VL series, InternVL series, or models with equivalent capabilities. These models require appropriate configuration to optimize their performance on the table recognition task. For example, by adjusting the model's input processing methods (such as image preprocessing) and task prompt design, it is ensured that the model can focus on the table's row and column information, cell content, and physical layout.

[0030] Multimodal large language models are advanced artificial intelligence models that combine the capabilities of computer vision (processing images, videos, etc.) and natural language processing (understanding and generating text). They can simultaneously understand and process visual information in images as well as related textual information, thus better performing tasks involving multiple data types, such as extracting text and structure from tabular images. For example, Qwen-VL and InternVL are representative of this type of model.

[0031] Leveraging the powerful cross-modal understanding capabilities of multimodal large language models, unstructured or semi-structured tabular data (especially images) is transformed into a structured HTML format that is initially understandable by machines. A machine-readable, universally applicable structured representation format is a data format that computer programs can easily understand and use, organizing information in a standardized way, rather than being a collection of pixels like an image. HTML (Hypertext Markup Language) is a commonly used structured format that uses tags to define the structure and content of a document, such as... This indicates the start of the table. Indicates a line, It represents a cell, is highly versatile, and is easy for subsequent programs to parse and process.

[0032] Upon receiving input data, if it is a table image, the image data along with a carefully designed prompt is fed into the multimodal language model. This explicitly instructs the model to recognize the table's physical structure, including locating the overall table boundaries, defining the main row and column grid structure, and accurately capturing the boundaries of each cell. Simultaneously, the multimodal language model is required to extract all text content within the cells, maintaining the accuracy of the original text as much as possible. For complex structures commonly found in tables, especially merged cells spanning multiple rows or columns, the model needs to pay special attention and use the `rowspan` and `colspan` attributes in the output HTML to correctly mark these spanning relationships, maintaining structural accuracy. Finally, the multimodal language model generates a preliminary HTML document, marked as (T0). This document aims to reproduce the structure and content of the original table as faithfully as possible.

[0033] If the input data is already partially structured, such as HTML fragments obtained from parsing LaTeX source code, the focus is on standardizing these fragments into a more unified and standard-compliant HTML format, or using the visual understanding capabilities of the multimodal large language model to further confirm and correct any possible structural deviations. The whole process is a transformation from unstructured / semi-structured to preliminary structured, relying on the powerful perception and generation capabilities of the multimodal large language model.

[0034] Table content recognition is the process by which a multimodal large language model analyzes table images or structured data to identify individual elements within the table. This includes recognizing table boundaries, rows, columns, cells, and the specific content contained within cells, such as text, numbers, and symbols. Structured processing involves the multimodal large language model organizing the identified table content according to a specific logic and format, transforming it into structured data that a machine can understand and process—that is, generating an HTML representation. The initial HTML structure representation is an intermediate result output by the multimodal large language model; it is an HTML document fragment containing the basic structural information of the table, such as... The label indicates the start of the table. Labels indicate rows. or Labels represent cells, the text content within those cells, and can also reflect the physical layout of the table, such as through nested labels. and The `<rowspan>` tag indicates row and column relationships, and the `rowspan` and `colspan` attributes indicate merged cells. However, this representation is preliminary and may contain errors or be inaccurate.

[0035] Table row and column information refers to the number of rows in the table as represented in HTML. The number of labels and the number of columns (same) Under the tag or The maximum number of tags (considering the impact of merged cells). Cell content is in HTML representation. or The text content between the tag pairs is the text, numbers, and other information extracted from the table by MLLM. Physical layout information is the visual arrangement of the table in the HTML representation, reflected by the nesting relationship of tags and attributes (such as rowspan, colspan), for example, which cells span multiple rows or columns.

[0036] Leveraging the powerful processing capabilities of multimodal large language models, the integrated initial input information is passed to a pre-selected multimodal large language model. Upon receiving the input, the multimodal large language model simultaneously processes visual information (if images are present) and textual information (context, and possibly semi-structured text). For the image portion, the visual encoder of the multimodal large language model analyzes image pixels, recognizing table outlines, grid lines, text regions, etc. For the text portion, the language model portion of the multimodal large language model understands the background information provided by the context, helping to interpret ambiguous or abbreviated terms in the table. Then, based on its internally trained knowledge, the multimodal large language model infers the table structure, including the number of rows and columns and cell boundaries. Next, it reads the text content within each cell and transcribes it as accurately as possible. In this process, the multimodal large language model pays special attention to complex cell structures, such as merged cells, analyzing which cells visually span multiple rows or columns and using the corresponding rowspan and colspan attributes to mark these merge relationships in its generated HTML output. Finally, the multimodal large language model organizes all this identified information into an HTML-formatted string, outputting it as the initial HTML structural representation (denoted as T0). This T0 file contains the basic framework of the table, the text content of all cells, and their physical layout information, such as their positions and merging relationships within the table.

[0037] For example, a table image displays the activity data of catalyst A at different temperatures. The initial input information includes the table image and the table title. Figure 1 The input includes the activity test results of catalyst A at different temperatures, and a description from the preceding text (to investigate the effect of temperature on the performance of catalyst A, activity tests were conducted at 100℃, 150℃, and 200℃...). Upon receiving this input, the multimodal large language model (e.g., Qwen-VL-32B-0414) first analyzes the image. The visual part identifies it as a 3x4 table. The first row is the header, containing temperature (℃), activity (%), reaction conditions, and remarks. The second and third rows are the data rows. The second row corresponds to the data at 100℃: activity 82%, reaction conditions at standard atmospheric pressure, and remarks NULL. The third row corresponds to the data at 150℃: activity 95%, reaction conditions the same as the second row, and remarks NULL. The multimodal large language model notices that the activity (%) column visually spans two data rows, therefore determining it to be a merged cell. Meanwhile, the contextual information about 100℃, 150℃, and 200℃, while only showing the first two temperature rows in the table image, hinted at the possible existence of a third row, 200℃. This affected the multimodal large language model's initial judgment of the table's completeness. However, based on the image content, the visible portion was primarily processed. The multimodal large language model successfully recognized the table's row and column structure (3 rows and 4 columns, considering merging), extracted key text (temperature value, activity percentage, reaction conditions, remarks), and correctly handled merged cells (rowspan=2 and colspan=3). The output T0 contained the table's row and column information (via...). and nested / (Reflection), cell content (e.g., 100, 82, standard atmospheres), and physical layout information (e.g., ... (Label's rowspan and colspan attributes). MLLM achieved 98% accuracy in identifying activity as activity (%), 99.5% accuracy in identifying values ​​of 82 and 95, and 97% accuracy in identifying standard atmospheric pressure. While the recognition rate is high, there is still approximately a 2% possibility of header recognition errors or numerical value misidentification, as well as the possibility of missing unclear annotations in the image (such as the activity data at 200°C in the third row being misidentified or ignored due to image unclearness).

[0038] Leveraging the powerful multimodal processing capabilities of a multimodal large language model, an automated conversion was achieved from unstructured or semi-structured tabular data sources (especially image formats that are difficult to parse directly) to machine-readable structured HTML representations with a certain degree of accuracy.

[0039] Iterative self-correction is performed based on the initial HTML structure representation, the original table image, and the table context information to output a high-fidelity HTML structure representation.

[0040] Further details are attached. Figure 2 As shown, this application further includes the following steps: for the content of each cell in the initial HTML structure representation, performing semantic alignment processing of cell content by combining logical position, original table image and table context information, and outputting a first corrected structured table; performing cross-cell content coherence and logical consistency correction on the first corrected structured table to generate a second corrected structured table; performing table physical structure and layout correction on the second corrected structured table to generate a third corrected structured table; performing semantic fidelity final check on the third corrected structured table, and outputting the high-fidelity HTML structure representation.

[0041] Furthermore, this application also includes the following steps: semantic alignment processing of cell content, including correction of character errors, improper word segmentation, and misrecognition of numbers and special symbols.

[0042] Furthermore, this application also includes the following steps: extracting cross-cell content from the first corrected structured table, specifically including cross-row and cross-column content organization; determining whether the cross-cell content has one or more of the following: logical breaks, misalignment, or semantic conflicts, and generating problematic cell content; correcting and adjusting the problematic cell content to generate the second corrected structured table.

[0043] Furthermore, this application also includes the following steps: table physical structure and layout correction, including matching the number of rows and columns, and correct identification and representation of merged cells.

[0044] Furthermore, this application also includes the following steps: semantic fidelity final inspection includes consistency correction between the content, structure, and the overall semantic level expressed by the combination of content and structure and the original table information.

[0045] Furthermore, this application also includes the following steps: the table context information includes the table's official title, a unique number or citation identifier in the literature, and narrative text paragraphs surrounding the table.

[0046] Specifically, to address potential recognition errors, content biases, or structural distortions in the structured table representations initially generated by multimodal large language models, a refined four-stage iterative self-correction mechanism driven by the Large Language Model (LLM) is introduced. This mechanism provides the LLM with a combined input including initial HTML, the original table image (as a visual reference), and contextual information, supplemented by specific task instructions. It guides the LLM to execute the following logical optimization steps, simulating the expert review and correction process of the table, rather than performing traditional numerical calculations or fixed algorithms. Multimodal large language models are a type of advanced artificial intelligence model whose core capability lies in their ability to collaboratively process and understand data from multiple different information channels (modalities), such as text, images, and sound, and to generate unified or specific modal outputs based on this multimodal fusion understanding. The primary task of the multimodal large language model is to initially convert the input table image into structured text (such as HTML). Large language models typically refer to deep neural network models pre-trained on large-scale text corpora through self-supervised learning and other methods. They possess excellent natural language understanding, text generation, logical reasoning, and context awareness capabilities. Large language models are the core enabling technologies for achieving self-correction of table content, recognition of scientific entities in specific domains, and inference of complex logical relationships (condition-outcome knowledge chains).

[0047] The initial HTML structure generated by the multimodal large language model is denoted as T0, the original table image is denoted as I, and the context information is denoted as M. The self-correction process aims to achieve this through an iterative optimization function F driven by the large language model. SC Generate a high-fidelity HTML structure representation (T4): T4 = F SC (T0,I,M), function F SC Internally, it contains four logical stages for application or iterative optimization. Each stage k∈{1,2,3,4} can be represented by a transformation function f. k :T k =f k (T k-1 ,I,M,P k ), where P k These are specific task prompts guiding the LLM operations at this stage. The transition from one stage to the next (T) k =f k (T k-1 ,I,M,P k ) is achieved by providing a tabular representation T containing the current stage to the Large Language Model (LLM). k-1 Original table image I, context information M, and specific task instructions P k The combined input is used to guide execution.

[0048] k∈{1,2,3,4} are T1, T2, T3, and T4 respectively. T1 is the first corrected structured table obtained after the first correction of the initial HTML structure representation T0; T2 is the second corrected structured table obtained after the second correction of the first corrected structured table T1; T3 is the third corrected structured table obtained after the third correction of the second corrected structured table T2; and T4 is the high-fidelity HTML structure representation obtained after the third correction of the third corrected structured table T3.

[0049] For each cell c in the initial HTML structure representation ij The initial extraction of the inner text, the goal of this stage is to generate T1, where each cell contains text′(c ij A more precise semantic representation is: text′(c ij ) = argmax text Sim sem (text,Visual(c ij ,I),Context(c ij ,M)), where, Sim sem It is a semantic similarity function, Visual(c ij ,I) represents the corresponding cell c in the original table image I. ij Visual features, Context(c ij (M) represents the contextual information associated with the cell. The goal is to maximize the consistency between the extracted text and the true semantics implied by visual and contextual cues.

[0050] Logical position refers to a cell's location within the table structure, typically indicated by row and column numbers. For example, a cell in the first row and first column, or a merged cell spanning multiple rows and columns. The original table image refers to the visual representation of the initially entered table, such as a scanned image or screenshot, containing the table's original visual details, such as font, size, color, and layout. Table context information is additional information related to the table data, including the table's official title, unique identifier or citation in the literature, and narrative text paragraphs surrounding the table. The table title briefly describes the table's content, helping to identify the table's subject and background; table citation information is the table's unique identifier or citation in the literature, used to locate the table; adjacent text paragraphs are the paragraphs surrounding the table, typically containing explanations of the table's content, experimental methods, or data interpretation. Utilizing the table's official title, citations, and adjacent text paragraphs as important contextual information helps correct cell content. For example, if the title mentions testing temperature (°C), then if a cell in the table is identified as 200, the context indicates that the value is temperature, not some other unit of measurement.

[0051] Based on the content of each cell, semantic alignment of cell content is performed by combining logical position, original table image, and table context information. This includes correction of character errors (e.g., changing "live" to "active"), improper word segmentation (e.g., incorrectly segmenting "significantly improve" into "obviously improve" and correcting it), number misidentification (e.g., misidentifying 82 as 32 and correcting it), and special symbol misidentification (e.g., misidentifying ± as + and correcting it). In short, the content of each cell in the structured table is examined one by one through a large language model. Combining its logical position in the table (row and column header information), visual feature cues of the corresponding area in the original image, and possible related explanations in the context text, deep semantic analysis and calibration are performed on the text within the cells. This corrects character errors, improper word segmentation, and misidentification of numbers or special symbols that may occur during the initial recognition by OCR or MLLM, ensuring that the semantic information carried by each cell is accurate. The output is a structured table representation with optimized content, denoted as the first corrected structured table.

[0052] The goal of cell content semantic alignment optimization is to maximize the semantic similarity between the extracted text of the cell and the true semantics inferred from the visual features and contextual information of the original table image. sem Review and calibrate the semantic accuracy of each cell's content, correcting any initially identified character errors, improper word segmentation, or misidentification of special symbols.

[0053] For the first corrected structured table T1, analyze and adjust the content organization across rows and columns in the table to restore and enhance the inherent logic and semantic coherence of the table data. coherence Generate the second corrected structured table. T2 = argmax T′ ·L coherence (T′∣T1), where L coherence It is a scoring function based on rule or model learning, which evaluates things such as the correspondence between table headers and data columns, and the consistency of data types within columns.

[0054] Based on the first corrected structured table T1, the language model further analyzes whether there are logical breaks, misalignments, or semantic conflicts in the organization of content across rows and columns. For example, it checks whether the table headers and data columns correspond correctly, and whether the data types remain consistent within a column (or conform to the expected pattern of change). The language model will attempt to adjust the content or interpretation of problematic cells to restore and enhance the inherent logic and semantic coherence of the table data. The output of this stage is recorded as the second corrected structured table.

[0055] Extract cross-cell content from the first corrected structured table, including cells spanning rows and columns. This content may contain important information in the table's physical layout (e.g., some headers may span multiple columns, or some data items may span multiple rows representing different experimental conditions). Check for one or more of the following: logical breaks, misalignments, or semantic conflicts. Logical breaks occur when cross-row or cross-column content is logically incoherent or inconsistent. Misalignments occur when cross-row or cross-column content is not arranged in the correct position, resulting in illogical cell arrangement and affecting the table's overall structure and readability. Semantic conflicts occur when cross-cell content semantically conflicts with data in other cells.

[0056] Examine the logical relationships between content across cells to determine if there are any logical mismatches or inconsistencies between table headings and data. For example, if an experimental condition heading spans multiple cells, but the content of these cells is inconsistent with the conditions described in the heading, this is identified as a logical break. Check if the content across cells is correctly arranged, especially for misalignments caused by crossing cells. For example, if the table header temperature spans multiple columns, but its subsequent cell positions are not correctly aligned, the temperature values ​​in the data rows are placed incorrectly. Through contextual and table content analysis, check for semantic conflicts between content across cells. For example, if some data in the temperature (°C) column of a table is incorrectly labeled as time (h), a semantic conflict is identified.

[0057] For any logical breaks, misalignments, or semantic conflicts discovered, the problematic cell content is marked and generated, and these cells are designated as problematic cells for subsequent correction. The problematic cell content is then corrected and adjusted. Based on the table context and other data, the logic of cross-cell content is adjusted to ensure consistency between the title and the data below it. If misalignments occur in the table, the content of cells spanning rows and columns is adjusted to ensure their position within the table matches the actual content. If semantic conflicts are found, the system corrects the content based on context and data type. After correcting the problematic cell content, a second corrected structured table is generated, resolving the cross-cell content issues in the original table and improving the table's logic, consistency, and accuracy.

[0058] For the second corrected structured table T2, based on the visual layout features and contextual information of the original table, the physical structure information in the structured representation, such as the number of rows, columns, and merged cells, is corrected. In other words, the physical structure and layout of the second corrected structured table are corrected to generate the third corrected structured table. Table physical structure and layout correction involves checking and adjusting the physical layout of the table (such as the number of rows and columns, and the merging of cells) during table processing to ensure that the table structure conforms to the expected design, including the correct identification and representation of the number of rows, columns, and merged cells. The focus is on correcting the HTML structure; let S(T) be the HTML structure of table T. The goal is to make S(T3) approximate the true physical structure S implied by the original image I and the context M as closely as possible. true (I,M): S(T3)=argmin S′ ·D struct (S′,S true (I,M)∣S(T2)), where D struct This is a distance function that measures structural differences, such as tree edit distance or accuracy assessment of rowspan / colspan attributes. The optimization objective for the table's physical structure and layout is to minimize the structural difference D between the corrected HTML structure and the true physical structure inferred from the original table image and contextual information. struct .

[0059] The language model focuses on the accuracy of the HTML structure represented by the second-corrected structured table T2, including precise matching of row and column numbers, and correct identification and representation of merged cells (such as the rowspan and colspan attributes in HTML). The language model will fine-tune and correct the HTML skeleton structure by referring to the visual layout features and contextual information (such as table dimensions or structure types implied in the title), ensuring it reflects the physical form of the original table. The output of this stage is denoted as the third-corrected structured table.

[0060] For the third corrected structured table T3, a comprehensive review and correction are conducted to address any remaining deviations between the structured representation and the original table information at the content, structure, and overall semantic levels. This ensures compliance with the preset structured representation specifications and maximizes its conformity (Conform(T,G)) with the predefined HTML structure specifications and semantic constraints G. T4 = argmax T′ ·Conform(T′,G∣T3).

[0061] The third-corrected structured table undergoes a final comprehensive review using a large language model to ensure that its content, structure, and overall semantic expression are highly consistent with the original table information and conform to predefined structured representation specifications (such as HTML standards). Any residual deviations are then corrected to output the final high-quality, high-fidelity structured table representation, denoted as the high-fidelity HTML structure representation.

[0062] Semantic fidelity final verification refers to the final validation of a table to ensure that the table data remains consistent with the original table information in terms of content, structure, and the combination of content and structure. Semantic fidelity means that the table's content, structure, and expression should faithfully reflect the true information conveyed by the original table as much as possible, without any ambiguity or misleading information. Content consistency means that the content of all cells in the table should be completely consistent with the data in the original table, including text, numbers, symbols, etc., without any misunderstanding or error. Structural consistency means that the physical structure of the table (such as rows, columns, and cell merging) should be consistent with the structure of the original table. That is, the arrangement and merging of each part of the table and each cell should accurately reflect the layout of the original table. The overall semantic meaning of the combination of content and structure means that the relationship between the content and structure in the table should effectively convey the overall semantic meaning of the original table. For example, column headers, row data, merged cells, etc., should jointly express the information that the table wants to convey through the correctness of the structure and the accuracy of the content. High-fidelity HTML structure representation, after semantic fidelity final verification, generates an HTML table structure that can restore the content and layout of the original table to the greatest extent possible. High fidelity means that the content, structure, and semantic expression of the table are completely consistent and can accurately convey all the information of the original table.

[0063] The four calibration stages can be iterated as needed until the output HTML quality converges or reaches a preset iteration limit. Through progressive calibration, a high degree of consistency in table content, layout, and semantics is ensured, avoiding logical errors and ambiguities. The generated HTML structure not only conforms to standards but also reflects all the information from the original table.

[0064] Based on a pre-defined scientific entity ontology, semantic information is extracted from the high-fidelity HTML structure representation, and the entity extraction results are output.

[0065] Furthermore, this application also includes the following steps: based on the preset scientific entity ontology, constructing first prompt information to guide the large language model to perform entity recognition; inputting the first prompt information into the large language model to drive the large language model to perform entity recognition tasks, performing deep semantic analysis on the high-fidelity HTML structure representation, and generating the entity extraction result.

[0066] Furthermore, this application also includes the following steps: a complete definition and detailed rules for the preset scientific entity ontology, used to clarify the scope and standards of entity recognition; task instructions, used to guide the large language model to perform entity recognition tasks in structured table content, and output the text span, category, and related metadata of each entity instance; the structured table content to be processed; contextual information related to the high-fidelity HTML structure, used to supplement semantic understanding; and at least one labeled entity extraction sample, used as an example to guide the large language model to understand the task requirements and output format.

[0067] Specifically, scientific entities are atomized information units identified from scientific literature (especially its tabular sections) that carry semantics of a specific scientific research field. A scientific entity ontology is defined, comprising seven basic categories: Problem, Method, Dataset, Material, Tool, Indicator, and Value. This aims to systematically and standardizedly annotate and extract semantics from tabular content. Each type of scientific entity has a clear semantic definition, typical examples, and identification criteria.

[0068] Based on a predefined scientific entity ontology, a first prompt is constructed to guide the large language model (which can be the same model or a different instance of the large language model used in the aforementioned iterative self-correction) in entity recognition. This first prompt includes: a complete definition and detailed rules of the predefined scientific entity ontology (providing detailed definitions and recognition rules for all entity categories to clarify the scope and standards of entity recognition); task instructions (to guide the large language model to perform entity recognition tasks within structured table content and output the text span, category, and related metadata for each entity instance); the structured table content to be processed (i.e., the structured table data from which entities are to be extracted); contextual information related to the high-fidelity HTML structure (including supplementary contextual information such as table titles, document numbers, and adjacent paragraphs to help the large language model understand the context of the entities); and at least one labeled entity extraction example (providing sample examples to demonstrate task requirements and output format, helping the large language model accurately understand these requirements and format). The large language model performs deep semantic analysis on the table content based on the first prompt to achieve high-precision entity extraction.

[0069] The initial prompt information is input into the large language model, driving it to perform entity recognition. This involves identifying which scientific entities are included in the predefined scientific entity ontology, performing deep semantic analysis on the high-fidelity HTML structure representation, identifying and extracting all scientific entities that meet the definition, and outputting the results according to a predetermined format. Let T4 be the self-corrected high-fidelity HTML structure representation. The scientific entity ontology is defined as O... SE = {E1, E2, ..., E7}, where E k This represents an entity category (such as a question, method, etc.). Entity extraction is a mapping function Φ. Extract Implemented by LLM: S Ent =Φ Extract (T4,M,O SE ,P Ent ), where P Ent This is a hint about entity extraction, S Ent It is the extracted set of entities, where each entity e∈S Ent Includes its text span, its category type(e)∈O SE and location information.

[0070] By combining scientific entity ontology definitions and task instructions, entities that conform to the definitions are identified from structured tables, ensuring that the identification of each entity is based on the context and logical structure of the table, thereby improving the accuracy of the extraction results.

[0071] Based on the preset logical paradigm and combined with the entity extraction results, the extracted entities are organized into a knowledge chain.

[0072] Furthermore, this application also includes the following steps: the preset logical paradigm is a condition-result knowledge chain model; a second prompting information is constructed to prompt the generation of knowledge chains; based on the second prompting information, logical reasoning and semantic analysis are performed on the entity extraction results to automatically identify and combine entities belonging to conditions and results, and construct multiple knowledge chains that meet the structural requirements, outputting a JSON list of knowledge chains containing multiple elements.

[0073] Specifically, the pre-defined logical paradigm is a structured model used to guide data organization and processing, namely the condition-outcome knowledge chain model. The condition-outcome knowledge chain is a structured knowledge representation paradigm designed for the characteristics of tabular data in scientific literature. It abstracts the scientific experiment or observation process described in the table into logical binary pairs consisting of two main elements: conditions and results. The condition part aggregates and describes information entities such as experimental premises, settings, input variables, and control parameters; the result part records information entities such as experimental outputs, observation data, and performance results triggered by these conditions, aiming to clearly and accurately capture the core logical relationships in the scientific process.

[0074] The condition-outcome knowledge chain model is used to highly condense and structure tabular information into logical binary groups consisting of conditional components (encapsulating input entities and their attributes such as experimental methods, objects of operation, environmental variables, and control parameters) and outcome components (recording output information such as corresponding performance indicators, observational data, and experimental phenomena). This clearly, systematically, and meticulously depicts the inherent logical relationship between experimental design elements and experimental outputs. In other words, the condition-outcome knowledge chain model is used to explicitly represent the core logical flow of scientific experiments or observational research. The conditional components encapsulate a series of entities and their attributes that constitute the experimental premise, input variables, control factors, or research background; the outcome components record the output variables, performance, experimental phenomena, or research conclusions observed under these conditions.

[0075] The second hint information for the knowledge chain reasoning and construction task of the large language model includes: detailed definition and construction rules of the condition-result knowledge chain model, high-fidelity HTML structure representation of the input and entity extraction results, relevant context information, and multiple completed condition-result knowledge chain examples.

[0076] Detailed definitions and construction rules for the condition-result knowledge chain model, such as how to determine which entity combinations should be classified into condition components and which should be classified into result components based on the table's row and column organization (e.g., variables defined in the table header and values ​​corresponding to data rows), and the type and content of entities within cells; how to handle complex situations with many-to-one or one-to-many condition-result mappings; how to identify and connect entity pairs that form logical chains; and how to effectively merge or distinguish chains with the same conditions but different result sub-items. Several completed condition-result knowledge chain examples are provided to standardize the LLM's output structure (e.g., outputting a JSON list where each JSON object represents a knowledge chain, containing explicit Condition and Result fields and their structured content).

[0077] The language model performs complex logical reasoning based on the second prompt, automatically extracting and constructing a set of clearly structured and semantically accurate condition-result knowledge chains from the table. Based on the second prompt, it performs logical reasoning and semantic analysis on the entity extraction results, comprehensively analyzing the table's logical structure, extracted entity information, and contextual semantics to automatically extract and construct a set of clearly structured and semantically accurate condition-result knowledge chains from the table.

[0078] The knowledge chain construction process can be formalized as follows: Let S Ent This represents the set of entities that have been extracted. Each knowledge chain... Represented as a pair K Ci =(Cond i Res i The construction process is an inference function Ψ executed by a large language model. KC : Among them, R KC These are the rules and hints for building a knowledge chain. (Cond) i ={e j ∈S Ent |IsCondition(e j ,T4,M,R kc )}.

[0079] Res i ={e k ∈S Ent |IsResult(e k ,T4,M,Cond i ,R kc )}.

[0080] Among them, IsCondition and IsResult are predicate logics that determine whether an entity belongs to a condition or result, and are reasoned by the large language model based on the table structure, entity type and contextual semantics.

[0081] For each identified condition and result, they are combined into a knowledge chain. When dealing with complex many-to-one or one-to-many relationships, multiple conditions may lead to the same result, or a single condition may lead to multiple different results. The final output is a set of structured condition-result knowledge chains (preferably in JSON format). Each knowledge chain explicitly represents a set of experimental conditions extracted from the input table and their corresponding results. Furthermore, intermediate processing results, such as high-fidelity HTML structure representations and entity extraction results, can also be selectively output as needed. The final output is a JSON list containing multiple knowledge chains, each accurately reflecting the relationship between experimental conditions and results in the table. Guided by the condition-result model and task instructions, the large language model can automatically identify and construct logically consistent knowledge chains from the table.

[0082] In summary, the method for generating knowledge chains for scientific and technological literature tables based on a large language model provided in this application has the following technical effects:

[0083] By receiving table images, extracted PDF regions, and structured source code from scientific literature, and combining them with table context information, initial input information is generated. A multimodal large language model is then invoked to perform table content recognition and structuring processing on the initial input information, generating an initial HTML structure representation, including table row and column information, cell content, and physical layout information. Iterative self-correction is performed based on the initial HTML structure representation, the original table image, and the table context information, outputting a high-fidelity HTML structure representation. Semantic information is extracted from the high-fidelity HTML structure representation based on a preset scientific entity ontology, outputting entity extraction results. Finally, according to a preset logical paradigm, the extracted entities are organized into a knowledge chain based on the entity extraction results. In other words, by receiving table data from scientific literature and integrating relevant context information, invoking a multimodal large language model for semantic processing to generate an initial HTML structure representation, iteratively self-correcting the initial HTML structure representation against the original table image and table context information to correct errors in the initial recognition process, performing semantic extraction based on a preset scientific entity ontology, and organizing a knowledge chain based on a preset logical paradigm, the accuracy of table content recognition and structure parsing is improved, thereby enhancing the accuracy of semantic recognition.

[0084] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0085] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating knowledge chains in scientific and technological literature tables based on a large language model, characterized in that, include: It receives table images, PDF extracted regions, and structured source code from scientific and technological literature, and generates initial input information by combining the table context information. The initial input information is processed by calling a multimodal large language model to perform table content recognition and structuring, generating an initial HTML structure representation, which includes table row and column information, cell content and physical layout information; Iterative self-correction is performed based on the initial HTML structure representation, the original table image, and the table context information to output a high-fidelity HTML structure representation. Based on a preset scientific entity ontology, semantic information is extracted from the high-fidelity HTML structure representation, and entity extraction results are output. Based on the preset logical paradigm and combined with the entity extraction results, the extracted entities are organized into a knowledge chain.

2. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 1, characterized in that, Iterative self-correction is performed based on the initial HTML structure representation, the original table image, and the table context information to output a high-fidelity HTML structure representation, including: For the content of each cell in the initial HTML structure representation, semantic alignment of cell content is performed by combining logical position, original table image and table context information, and the first corrected structured table is output. The first corrected structured table is corrected for cross-cell content coherence and logical consistency to generate a second corrected structured table; The physical structure and layout of the second corrected structured table are corrected to generate a third corrected structured table; A semantic fidelity final check is performed on the third corrected structured table, and the high-fidelity HTML structure representation is output.

3. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 2, characterized in that, Cell content semantic alignment processing includes correction of character errors, improper word segmentation, and misrecognition of numbers and special symbols.

4. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 2, characterized in that, The first corrected structured table is corrected for cross-cell content coherence and logical consistency to generate a second corrected structured table, including: Extract cross-cell content from the first corrected structured table, specifically including content organization across rows and columns; Determine whether the cross-cell content has one or more of the following: logical breaks, misaligned arrangement, or semantic conflicts, and generate the content of the problematic cell. The content of the problematic cells is corrected and adjusted to generate the second corrected structured table.

5. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 2, characterized in that, Table physical structure and layout correction includes matching the number of rows and columns, as well as the correct identification and representation of merged cells.

6. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 2, characterized in that, The semantic fidelity final check includes the consistency correction between the content, structure, and the overall semantic level expressed by the combination of content and structure and the original table information.

7. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 1, characterized in that, The table context information includes the table’s official title, its unique number or citation identifier in the literature, and the narrative text paragraphs surrounding the table.

8. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 1, characterized in that, Based on a pre-defined scientific entity ontology, semantic information is extracted from the high-fidelity HTML structure representation, and the entity extraction results are output, including: Based on the preset scientific entity ontology, a first prompt message is constructed to guide the large language model to perform entity recognition. The first prompt information is input into the large language model, which drives the large language model to perform entity recognition tasks, performs deep semantic analysis on the high-fidelity HTML structure representation, and generates the entity extraction result.

9. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 8, characterized in that, The first prompt message includes: The complete definition and detailed rules of the preset scientific entity ontology are used to clarify the scope and standards of entity recognition; Clear task instructions are provided to guide the large language model to perform entity recognition tasks in structured table content and output the text span, category, and related metadata of each entity instance; The structured table content to be processed; Contextual information related to the high-fidelity HTML structure is used to supplement semantic understanding; At least one labeled entity extraction example is used as a guide for the large language model to understand the task requirements and output format.

10. The method for generating knowledge chains of scientific and technological literature tables based on a large language model as described in claim 1, characterized in that, Based on a pre-defined logical paradigm and the entity extraction results, the extracted entities are organized into a knowledge chain, including: The preset logical paradigm is a condition-result knowledge chain model; Construct a second prompt message to aid in the generation of the knowledge chain; Based on the second prompt information, logical reasoning and semantic analysis are performed on the entity extraction results to automatically identify and combine entities belonging to conditions and results, and construct multiple knowledge chains that meet the structural requirements. The output is a JSON list of knowledge chains containing multiple elements.

Images