Hierarchical evidence retrieval and context compression method and system based on field reduction

Abstract

The present application provides a hierarchical evidence retrieval and context compression method and system based on field specification, relating to the field of natural language processing, comprising: obtaining a field set to be extracted to generate a field binding rule constraint; parsing the source document to obtain table header nodes, data cells and narrative text, constructing a hierarchical table header index, obtaining an attribution table header set and a sentence index; matching and searching the attribution table header set with a keyword set and an interference word set respectively to obtain compressed candidate contexts; generating candidate value texts, positioning the value interval and determining whether the position constraint is satisfied, taking the candidate value texts that satisfy the position constraint as field values; normalizing the field values; extracting the positioning relationship of the value interval relative to the attribution table header set and writing it into the cache, and rewriting the field binding rule specification according to the recurrence frequency of the failure mode when the position constraint is not satisfied. The present application reduces resource occupation during long document processing and reduces the mismatch between values and evidence.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a hierarchical evidence retrieval and context compression method and system based on field specification. Background Technology

[0002] There are generally two approaches to extracting information from long documents based on large language models. One approach is to directly input the entire text into the model for end-to-end extraction. When the document is long, inference latency and computational cost increase, and the model is also prone to target information location offsets or omissions in extremely long contexts.

[0003] Another approach is to first retrieve candidate contexts based on semantic similarity, compress them, and then feed them into the model for extraction. After extraction, substring matching is used to verify whether the extracted values ​​are fragments of the original text. This method can reduce the input size, but substring verification can only confirm whether the value text appears in the candidate context; it cannot determine whether the value has the required positional relationship with explanatory information such as table headers, units, and negation statements. When there are multiple similar values ​​or multiple columns with similar structures in the same candidate set, mismatches between values ​​and evidence are likely to occur. Existing compression methods often rely on overall semantic similarity or chapter matching when deciding which candidate context to include, failing to distinguish whether the evidence types required for different fields have been covered, and struggling to identify false sufficiency caused by the simultaneous retention of evidence fragments belonging to different data entities when multiple similar fields coexist in the same document. Existing negation detection methods often rely on linear windows or dependency syntax to define the scope, without combining it with table header attribution judgments. As a result, field-level extraction of long documents still commonly suffers from latency and resource waste, loss of target information, and inconsistencies between output values ​​and evidence.

[0004] Chinese invention patent application number 202311546547.7 discloses a method and system for compressing context based on a large language model. The method obtains the text to be compressed, adds task descriptions, delimiters, and compression slots; when computing power is limited, it freezes the large language model and trains the projection layer, mapping the encoded hidden layer states to compressed virtual characters; when computing power is sufficient, it pre-trains and compresses the large language model itself; and then generates a text response based on the compressed virtual characters and the uncompressed context. However, after compressing the context into virtual characters, it cannot trace back to the original text position, and it is difficult to verify whether the generated content has a consistent relationship with the explanatory information in the original text. Summary of the Invention

[0005] In view of this, the present invention provides a hierarchical evidence retrieval and context compression method and system based on field specification, which solves the technical problems in the prior art such as high inference latency and computing power consumption, easy location deviation or omission of target information, and simultaneous retention of evidence fragments belonging to different data entities when multiple similar fields coexist in the same document, when the full text of a long document is input into the model for extraction.

[0006] The technical solution of this invention is implemented as follows: On the one hand, this invention provides a hierarchical evidence retrieval and context compression method based on field specification, including: S1. Obtain the set of fields to be extracted from the source document, and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set, the interference word set, and the positional constraint between the evidence and the value corresponding to the current field. The interference word set is obtained by taking the union of the keyword sets of all fields in the set of fields to be extracted except the current field. The positional constraint is used to limit the relative positional relationship that the evidence and the value should satisfy. S2. Parse the source document to obtain header nodes, data cells, and narrative text. Construct a hierarchical header index based on ancestor header nodes on the list header path and row header path. Each data cell in the hierarchical header index corresponds to a set of ancestor header nodes, which serve as the header set to which the data cell belongs. Divide the narrative text into clauses and record the adjacency relationships between clauses to obtain a clause index. S3. Match the belonging header set with the keyword set and the interference word set respectively. For table columns in the belonging header set that match the keyword set but not the interference word set, retain their header nodes and values. For table columns in the belonging header set that match the interference word set, regardless of whether they also match the keyword set, only retain the header nodes and remove the values. Table columns in the belonging header set that do not match either the keyword set or the interference word set are not retained. Match the narrative text with the keyword set and the interference word set according to the sentence index, and retain the corresponding matched content. Merge the retained table portion with the narrative text portion to obtain the compressed candidate context. S4. Input the compressed candidate context into the generation model to obtain candidate value text; search for a continuous string that is consistent with the candidate value text in the compressed candidate context, and locate the position of the searched string as the value range of the candidate value text in the compressed candidate context. S5. Determine whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint. The evidence includes at least one of the following: a set of attribution headers, unit words, and negation trigger words. When the position constraint is satisfied, the field values ​​corresponding to the value range are extracted and normalized to obtain the structured extraction result; the positioning relationship of the value range relative to the set of the belonging table header is recorded and written to the cache; If the position constraint is not met, the extraction is marked as a failure and the failure mode is recorded. The failure mode is statistically analyzed according to the frequency of recurrence. When the frequency of recurrence reaches a preset threshold, a specification patch is generated and the field binding specification is written back.

[0007] Based on the above technical solutions, preferably, the step of generating a field binding specification for each field in the set of fields to be extracted specifically includes: Generate a keyword set based on the name and synonyms of the table header corresponding to the current field; The set of keywords corresponding to the headers of all fields in the field set to be extracted, excluding the current field, is merged into the set of pre-defined explicit antonyms to obtain the set of interference words. The explicit antonyms are taken only from the antonyms of the headers already recorded in the field set to be extracted and are not introduced from external knowledge bases. Write the set of mandatory evidence types for the current field in the field binding specification. The mandatory evidence types are selected from the table header, unit words, negative trigger words, and past trigger words. Write a time window in the field binding specification, wherein the time window is determined by the in-text time expression; The binding location category set is written in the field binding specification. The binding location category is a non-empty subset of the row, column, sentence, character window, and mutually exclusive location. When the binding location category includes both table row and column categories and character window, table row and column categories take precedence over character window. The binding location category set serves as a positional constraint between evidence and value. Write assertion subtypes in the field binding specification. Assertion subtypes include measurement type and diagnostic type. The field value corresponding to the measurement type is the measurement value, and the field value corresponding to the diagnostic type is the noun text.

[0008] Based on the above technical solutions, preferably, each evidence type in the set of mandatory evidence types is written into a supplementary strategy, and the supplementary strategy includes a supplementary order and a maximum number of supplementary attempts. The order of insertion is as follows: the first priority is the column in the same table that is adjacent to the current search position and whose header set does not match the set of interfering words; the second priority is the clause before the current clause; the third priority is the column in the same header set whose header set matches the set of keywords. When evidence that meets the requirements is found according to the order of insertion, the priority search is stopped; the upper limit of the number of insertions is used to limit the maximum number of searches that can be performed on a single field during the insertion process, and the insertion is terminated after the upper limit of the number of insertions is reached.

[0009] Based on the above technical solutions, preferably, step S2 specifically includes: Identify table areas and narrative text areas in the source document; For the header cells in the table area, build a header path tree according to the inclusion relationship from the outermost to the innermost layer. The node sequence from the root node to the leaf node constitutes the table header path and the row header path, respectively. For each data cell, backtrack along the column and row directions on the header path tree, and form the ancestor header nodes on the backtracking path into a set of belonging headers; The narrative text area is segmented into clauses according to punctuation, and the adjacency relationship between clauses is recorded to obtain the clause index.

[0010] Based on the above technical solutions, preferably, the step of backtracking along the column and row directions of each data cell in the table header path tree specifically includes: The innermost header node of the column containing the data cell is used as the column starting point, and the innermost header node of the row containing the data cell is used as the row starting point. Starting from the column, proceed along the column header path to the root node and take parent nodes layer by layer. Starting from the column, proceed along the row header path to the root node and take parent nodes layer by layer. Include the header nodes that are backtracked into the parent header set. Other child nodes under the same parent node that do not appear on the backtracking path are not included in the parent header set. When the header node on the backtracking path is a merged cell, the merged cell is included in the parent header set as the same ancestor node shared by multiple header paths or row header paths, and the data cells covered by the same merged cell point to the same ancestor header node.

[0011] Based on the above technical solutions, preferably, step S3 specifically includes: Calculate the coherence of data cells in the hierarchical header index: ; Where π represents the data cell in the hierarchical header index, and f represents the current field. For header coherence, For header exclusivity, For unit satisfiability, The relevance of data cell π to the current field f; Calculate coherence for each data cell in the hierarchical header index. ,Will The column containing the data cell with a value of 1 should retain its header node and value completely. The column containing cells with a value equal to 1 retains only the header nodes and removes the values; if all data cells... If none of the values ​​are equal to 1, then it is determined that there are no relevant data cells in the current field; If no relevant data cells exist, first fill in the table with columns adjacent to the current search position that do not match the interference word set in the header set; if no relevant data cells still exist after filling in, then fill in columns with the same header set that match the keyword set in the header set; in the narrative text, fill in the clause preceding the current clause, and recalculate each time it is filled in. Stop when the maximum number of replenishments is reached; Based on the clause index, clauses that match the keyword set and their adjacent clauses are retained, while other clauses are not retained. The retained narrative text is arranged after the table columns retained by the matching retrieval and incorporated into the compressed candidate context.

[0012] Based on the above technical solutions, preferably, step S4 specifically includes: A generative model is used to generate candidate value text in the compressed candidate context, without using the character position output by the generative model as the value range; Search for a continuous string that is completely consistent with the candidate value text in the compressed candidate context. Each time a match is found, the start and end positions of the match are recorded as a value interval. If no match is found, it is determined that the candidate value text cannot be traced back to the source document and the extraction is marked as failed. If one or more value intervals are found, proceed to step S5.

[0013] Based on the above technical solutions, preferably, determining whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint specifically includes: When the value range originates from a table, the set of headers corresponding to the data cell containing the value range is read; when the value range originates from narrative text, the clause containing the value range is read; the positional relationship between the set of headers, unit words, and negative trigger words relative to the value range is checked, and the actual established positional relationship is selected according to the recorded order of positional constraints in the field binding specification, and the association determination result is calculated; wherein the positional relationship includes row, column, sentence, or character window: ; in, Indicates the range of values The association determination result is 1, which indicates that the value range passes the association determination, and 0, which indicates that the association determination fails. v To determine the character span for the value, For the coherence criteria of the table header, As a unit criterion, To exclude criteria; Will The range of values ​​is used as a candidate range for association; if there is only one candidate range for association, it is determined that the range of values ​​satisfies the positional constraint. When multiple candidate intervals exist that are linked, they are selected in the following order of priority: Value ranges derived from tables take precedence over value ranges derived from narrative text; table header constraints are ranked from highest to lowest; the number of constraints that are met is ranked from highest to lowest; character distance is ranked from lowest to highest; reading order is ranked from front to back; if the distinction cannot be made by priority, the ambiguity is marked as unsatisfactory and the position constraint is determined to be unmet. When there is only one associated candidate interval or one is selected according to priority, the value interval is determined to satisfy the positional constraint.

[0014] Based on the above technical solutions, the preferred one is... When the position constraint is satisfied in step S5, it specifically includes: When the evidence corresponding to the value range contains unit words, the unit conversion and format normalization of the field values ​​corresponding to the value range are performed based on the unit words to obtain normalized field values; when the evidence corresponding to the value range does not contain unit words, the field values ​​corresponding to the value range are directly used as normalized field values. The in-text time expression recorded in the field binding specification is read as a time window, and it is determined whether the normalized field value falls within the time window; the normalized field value that falls within the time window is used as the field value that passes the verification, and the value that does not fall within the time window is not entered into the structured processing; the in-text time expression is not written as an independent field in the structured extraction result. The validated field values, the value ranges, the hit header nodes and negative trigger words and their corresponding positions, whether they come from tables or narrative text, and the version number of the field binding specification are organized into a structured extraction result. Read the validated field values ​​and corresponding value ranges from the structured extraction results, extract the positioning relationship of the value range relative to the set of the belonging table header, as well as the table header node, unit word, and negative trigger word, and write it to the cache; when processing new source documents with similar layouts, narrow the search range of candidate data cells according to the positioning relationship in the cache; regardless of whether the cache is hit, the value range is repositioned and the position constraint is re-determined, and hitting the cache does not replace the position constraint determination.

[0015] More preferably, when the position constraint in step S5 is not satisfied, it specifically includes: Count the failure modes that are not written into the structured extraction results. The failure modes include any of the following: string retrieval failure, irrelevant table header, exclusion criterion trigger, correct column not entering the candidate context, ambiguity failure, and time window not falling into the target area. When the count of any failure mode reaches a preset threshold, a specification patch is generated and the keyword set, interference word set, or negative trigger word set is written back.

[0016] In addition, the present invention also provides a hierarchical evidence retrieval and context compression system based on field reduction, to implement the above-mentioned hierarchical evidence retrieval and context compression method based on field reduction, including: The field binding module is used to obtain the set of fields to be extracted from the source document and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set of the table header corresponding to the current field, the interference word set, and the positional constraints between the evidence and the value. The interference word set is the union of the keyword sets of the table headers of all fields in the set of fields to be extracted except the current field. The index building module is used to parse the source document to obtain the header nodes, data cells and narrative text. It constructs a hierarchical header index from the ancestor header nodes on the list header path and row header path, and obtains the set of the belonging header and the sentence index. The retrieval compression module is used to match the belonging header set with the keyword set and the interference word set respectively. The table column that matches the keyword set retains the complete header node and data cell. The table column that matches the interference word set only retains the header node and removes the values. When it matches both the keyword set and the interference word set, the interference word set is used as the standard to obtain the compressed candidate context. The backtesting module is used to generate candidate value text in the compressed candidate context using a generative model, retrieve a continuous string in the compressed candidate context that is completely consistent with the candidate value text, locate the value interval, determine whether the value interval satisfies the position constraint, and take the candidate value text that satisfies the position constraint as the field value. The normalization module normalizes the field values ​​that meet the position constraints based on the hit unit words, generating structured extraction results. The path caching module is used to extract the positioning relationship of the value range relative to the set of the belonging table header from the structured extraction results, write it into the cache, and write back the field binding specification according to the recurrence frequency of the failure mode when the position constraint is not met.

[0017] The present invention has the following advantages over the prior art: (1) A keyword set, interference word set, and position constraints are generated in advance for each field to be extracted. The table columns are distinguished and retained according to the hierarchical table header index, and the narrative text is matched and retained according to the sentence index before extraction and position verification. The final output is a structured extraction result that can be traced back to the original text position. Compared with the processing method of first compressing the candidate context as a whole and then directly accepting the model output, this invention puts the definition of the evidence scope before extraction and adds position verification after the value is produced. This allows the reduction of the input scale and the traceability and verifiability of the value to be taken into account at the same time. Overall, it is beneficial to reduce the resource consumption when processing long documents and reduce the mismatch between the value and the evidence.

[0018] (2) Distinguishing and retaining table columns by keyword set and interference word set helps reduce the probability of misjudging the value of adjacent column as the value of the current field in the extraction process, and correspondingly reduces the difficulty of determining the source of evidence in the scenario where similar fields coexist.

[0019] (3) The set of the belonging header is composed of the ancestor nodes on the list header path and the row header path. During the backtracking process, other child nodes under the same parent node are not included. The merged cells are treated as shared ancestor nodes to reduce the false association caused by inaccurate identification of the header hierarchy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the hierarchical evidence retrieval and context compression method based on field specification of the present invention; Figure 2 This is a flowchart illustrating the position constraint determination process of the hierarchical evidence retrieval and context compression method based on field specification of the present invention. Figure 3 This is a system framework diagram of the hierarchical evidence retrieval and context compression method based on field specification of the present invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, this invention provides a hierarchical evidence retrieval and context compression method based on field specification, comprising: S1. Obtain the set of fields to be extracted from the source document, and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set, the interference word set, and the positional constraint between the evidence and the value corresponding to the current field. The interference word set is obtained by taking the union of the keyword sets of all fields in the set of fields to be extracted except the current field. The positional constraint is used to limit the relative positional relationship that the evidence and the value should satisfy.

[0024] The source document can be a long, unstructured document such as a clinical trial report, medical record, or insurance claim material, containing both tables and narrative paragraphs. The set of fields to be extracted is provided in advance by the extraction task. For example, the fields to be extracted might include fasting blood glucose and admission diagnosis.

[0025] Specifically, a field binding specification is generated for each field in the set of fields to be extracted, including: Generate a keyword set based on the name and synonyms of the table header corresponding to the current field; The set of keywords corresponding to the headers of all fields in the field set to be extracted, excluding the current field, is merged into the set of pre-defined explicit antonyms to obtain the set of interference words. The explicit antonyms are taken only from the antonyms of the headers already recorded in the field set to be extracted and are not introduced from external knowledge bases. Write the set of mandatory evidence types for the current field in the field binding specification. The mandatory evidence types are selected from the table header, unit words, negative trigger words, and past trigger words. A time window is written in the field binding specification. The time window is determined by the in-text time expression. The in-text time expression is used to limit the time range in which the field value should fall, and is not written as a field to be extracted into the structured extraction result. The binding positioning category set is written in the field binding specification. The binding positioning category is a non-empty subset of the following: same row, same column, same sentence, character window, and mutual exclusion positioning. When the binding positioning category includes both table row and column categories and character window, table row and column take precedence over character window. The binding positioning category set serves as a positional constraint between evidence and value. Among them, same row means that the value range and evidence are located in the same row of the table, same column means that they are located in the same column of the table, same sentence means that they are located in the same clause of the narrative text, and character window means that the character offset between the evidence and the value range does not exceed the window length preset by the specification. Mutually exclusive positioning means that the value range of the current field must not fall into the same data cell, the same table column, or the same clause as the value range of the competing field recorded in the specification. Write assertion subtypes in the field binding specification. Assertion subtypes include measurement type and diagnostic type. The field value corresponding to the measurement type is the measurement value, and the field value corresponding to the diagnostic type is the noun text.

[0026] Understandably, diagnostic fields correspond to noun fields such as diagnosis names, and negation is applied directly to the value text.

[0027] This invention distinguishes and retains table columns according to keyword sets and interference word sets, so that columns containing competing fields that are adjacent in layout and similar in semantics are separated in the compression stage. The candidate context will not retain comparable values ​​belonging to different columns at the same time, which helps to reduce the probability of misjudging the value of the adjacent column as the value of the current field in the subsequent extraction stage, and correspondingly reduces the difficulty of determining the source of evidence in the scenario where similar fields coexist.

[0028] In one embodiment of the present invention, fasting blood glucose, postprandial blood glucose, glycated hemoglobin, and admission diagnosis are extracted simultaneously from the same source document. For the fasting blood glucose field, a keyword set is first generated from the header name and synonyms, taking fasting blood glucose, fasting GLU, and FBG. The keyword sets of the remaining fields are merged and added to the explicit antonyms predefined in the specification to obtain the interference word set. The keyword set of the postprandial blood glucose field includes postprandial blood glucose and PBG, and the keyword set of the glycated hemoglobin field includes glycated hemoglobin and HbA1c; both are added to the interference word set of fasting blood glucose. The fasting blood glucose field also lists "postprandial" and "random" as explicit antonyms. Explicit antonyms are only taken from the header antonyms already recorded within the set of fields to be extracted, and are not introduced from external thesaurus or medical knowledge bases. When the admission diagnosis and fasting blood glucose are not adjacent in layout, the keyword set of the admission diagnosis will still be entered into the interference word set of fasting blood glucose according to step S1; because the two fields are not adjacent in layout, the compression matching will usually not accidentally affect the relevant columns of the current field. When the set of fields to be extracted contains only the current field, the interference word set is empty.

[0029] In one embodiment of the present invention, the unit word type corresponds to dimensional terms, such as mmol / L, mmHg, %. The negation trigger word type corresponds to denial, not seen, none, not found, etc. The past trigger word type corresponds to past, past history. The in-text time expression type corresponds to time phrases such as admission time, examination day, and a certain year, month, and day.

[0030] In one embodiment of the present invention, the keyword set is generated based on the name and synonyms of the header corresponding to the current field. The generation method is as follows: read the header name recorded in the field definition, and write the header name and its pre-listed synonyms into the keyword set. For example, the keyword set for the fasting blood glucose field includes fasting blood glucose, fasting, and FPG. The keyword set for the hospital admission diagnosis field includes, for example, hospital admission diagnosis, preliminary diagnosis, and diagnosis. During matching, the header node or clause text is segmented to obtain a vocabulary set, which is then intersected with the keyword set. A non-empty intersection is considered a match. Synonyms are only used based on entries pre-written into the keyword set; the matching process does not perform semantic expansion or query external knowledge bases.

[0031] In one embodiment of the present invention, the interference word set is constructed as follows: the keyword sets corresponding to the headers of all fields in the field set to be extracted (excluding the current field) are merged, and then a pre-defined explicit antonym is added. Entries from medical ontology or external thesaurus are not included in the interference word set. The interference word set is empty when the field set to be extracted contains only the current field, or when the specification states that the current field and other fields have no adjacent layout competition. In this case, the matching search only uses the keyword set and does not trigger silent competing columns.

[0032] In one embodiment of the present invention, each evidence type in the mandatory evidence type set is written into a supplementation strategy, the supplementation strategy including a supplementation order and a maximum number of supplementations; The order of insertion is as follows: the first priority is the column in the same table that is adjacent to the current search position and whose header set does not match the set of interfering words; the second priority is the clause before the current clause; the third priority is the column in the same header set whose header set matches the set of keywords. When evidence that meets the requirements is found according to the order of insertion, the priority search is stopped; the upper limit of the number of insertions is used to limit the maximum number of searches that can be performed on a single field during the insertion process, and the insertion is terminated after the upper limit of the number of insertions is reached.

[0033] Specifically, the fasting blood glucose field typically includes a header, unit term, and negative trigger term in the specification. The admission diagnosis field typically includes a header and negative trigger term, but not a unit term. Evidence types not included in the specification are not subject to mandatory coverage determination.

[0034] S2. Parse the source document to obtain header nodes, data cells, and narrative text. Construct a hierarchical header index based on ancestor header nodes on the list header path and row header path. Each data cell in the hierarchical header index corresponds to a set of ancestor header nodes, which serve as the header set to which the data cell belongs. Divide the narrative text into clauses and record the adjacency relationships between clauses to obtain a clause index. Specifically, step S2 includes: Identify table areas and narrative text areas in the source document; For the header cells in the table area, construct a header path tree according to the inclusion relationship from the outermost to the innermost layer. The node sequence from the root node to the leaf node constitutes the list header path and the row header path, respectively: ; in, Represents the header cell A header path from the root node to a leaf node. For the header cell, This is the outermost header identifier. This is the innermost header identifier. The depth of the list header tree; For each data cell, backtrack along the column and row directions on the header path tree, and form the ancestor header nodes on the backtracking path into a set of belonging headers; The narrative text area is segmented into clauses according to punctuation, and the adjacency relationship between clauses is recorded to obtain the clause index.

[0035] Understandably, the table area is first identified by grid lines, cell coordinates, or table markers, and the remaining continuous text is treated as the narrative text area.

[0036] The present invention assigns a header set based on the ancestor nodes on the list header path and row header path. During the backtracking process, other child nodes under the same parent node are not included, and merged cells are treated as shared ancestor nodes, thereby reducing erroneous associations caused by inaccurate header hierarchy identification.

[0037] In one embodiment of the present invention, backtracking along the column and row directions of the table header path tree for each data cell specifically includes: The innermost header node of the column containing the data cell is used as the column starting point, and the innermost header node of the row containing the data cell is used as the row starting point. Starting from the column, proceed along the column header path to the root node and take parent nodes layer by layer. Starting from the column, proceed along the row header path to the root node and take parent nodes layer by layer. Include the header nodes that are backtracked into the parent header set. Other child nodes under the same parent node that do not appear on the backtracking path are not included in the parent header set. When the header node on the backtracking path is a merged cell, the merged cell is included in the parent header set as the same ancestor node shared by multiple header paths or row header paths, and the data cells covered by the same merged cell point to the same ancestor header node.

[0038] In one embodiment of the present invention, the source document is divided into non-overlapping logical partitions, and each narrative clause and each data cell is assigned a partition identifier and a channel identifier, specifically including: Logical partitioning is based on layout headings, a pre-defined partition thesaurus, and table boundaries. The pre-defined partition thesaurus includes at least the following for medical records: past medical history, present medical history, personal history, family history, physical examination, auxiliary examinations, and admission diagnosis; and for forms: first-level section headings. Each descriptive clause and each data cell is assigned a partition identifier. A table as a whole belongs to the partition corresponding to its chapter. Tables continuing across pages inherit the partition identifier from the first page table. When a document has no available headings, it is divided into continuous paragraph blocks. Adjacent paragraphs not interrupted by blank lines, page breaks, or tables are assigned to the same partition. Once partition boundaries are defined, exclusionary evidence such as negation or past evidence cannot cross these boundaries to affect values ​​in other partitions. For any character span, its channel is determined by whether it falls within both the row and column identifier ranges of a data cell. If both fall within the range, it is classified as a table channel; otherwise, it is classified as a descriptive channel.

[0039] A specific embodiment will be used for illustration: Taking a test table as an example, the first row merges cells to contain blood glucose, covering columns 2 and 3. The second row has column 2 as the header for fasting and column 3 as the header for postprandial. The third row has column 1 as the item name "glucose," column 2 as the data cell with value 5.6, and column 3 as the data cell with value 8.2. For data cell 5.6, the column-wise starting point is the innermost column header "fasting," and the row-wise starting point is the innermost row header "glucose." Tracing back along the column header path, passing through fasting and blood glucose, and tracing back along the row header path, passing through glucose, the resulting header set is blood glucose, fasting, and glucose. Another child node under the same parent node "blood glucose," "postprandial," does not appear on the tracing path and is not included in the data cell's header set. For data cell 8.2, the header set is blood glucose, postprandial, and glucose, excluding fasting. In this way, the fasting and postprandial columns will not be classified as the same column simply because their outermost column is also "blood glucose."

[0040] This invention only includes the ancestor table header in the scope of attribution. Table headers at the same level will not be merged into the same set of attribution headers. Parallel items such as fasting and post-meal in the cross table will not be misjudged as the same column.

[0041] S3. Match the belonging header set with the keyword set and the interference word set respectively. For table columns in the belonging header set that match the keyword set but not the interference word set, retain their header nodes and values. For table columns in the belonging header set that match the interference word set, regardless of whether they also match the keyword set, only retain the header nodes and remove the values. Table columns in the belonging header set that do not match either the keyword set or the interference word set are not retained. Match the narrative text with the keyword set and the interference word set according to the sentence index, and retain the corresponding matched content. Merge the retained table portion with the narrative text portion to obtain the compressed candidate context. In one embodiment of the present invention, the attribution header set is matched and retrieved with the keyword set and the interference word set, specifically including: Calculate the coherence of data cells in the hierarchical header index: ; Where π represents the data cell in the hierarchical header index, and f represents the current field. For header relevance, the value is 1 if the intersection of the header set to which the data cell π belongs and the keyword set of the current field f is not empty; otherwise, it is 0. For header exclusion, the value is 1 if the intersection of the header set corresponding to data cell π and the interference word set of the current field f is not empty, and 0 otherwise. For unit satisfiability, when the field binding specification records the unit term type as a mandatory evidence type, if the unit term exists in the adjacent column or preset neighborhood of the data cell π, the value is 1; otherwise, it is 0. When the field binding specification does not record the unit term type as a mandatory evidence type... Always takes 1, This represents the relevance of data cell π to the current field f, with 1 indicating relevance and 0 indicating incoherence. Calculate coherence for each data cell in the hierarchical header index. ,Will The column containing the data cell with a value of 1 should retain its header node and value completely. The column containing cells with a value equal to 1 retains only the header nodes and removes the values; if all data cells... If none of the values ​​are equal to 1, then it is determined that there are no relevant data cells in the current field; If no relevant data cells exist, first fill in the table with columns adjacent to the current search position that do not match the interference word set in the header set; if no relevant data cells still exist after filling in, then fill in columns with the same header set that match the keyword set in the header set; in the narrative text, fill in the clause preceding the current clause, and recalculate each time it is filled in. Stop when the maximum number of replenishments is reached; Based on the clause index, clauses that match the keyword set and their adjacent clauses are retained, while other clauses are not retained. The retained narrative text is arranged after the table columns retained by the matching retrieval and incorporated into the compressed candidate context.

[0042] Understandably, retaining the header of the competing field is to enable extraction to identify the existence of adjacent similar fields and thus correctly classify the header of the current field, while removing its value is to prevent the value of the competing field from being mistakenly identified as the value of the current field.

[0043] For each table column, the following three branches are applied: If the column matches both the header set and the keyword set but not the interference word set, the header node and data cells are fully retained; if the column matches both the header set and the interference word set, only the header node is retained and its values ​​are removed; if the column matches both the keyword set and the interference word set, it is processed according to the interference word set branch. If the column matches neither the keyword set nor the interference word set, the entire column is discarded and not included in the compressed candidate context. Based on the clause index, clauses matching the keyword set and their adjacent clauses are retained, while other clauses are not retained. The retained narrative text is arranged after the table columns retained by the matching retrieval and incorporated into the compressed candidate context.

[0044] This invention removes irrelevant columns and irrelevant clauses by column before values ​​are generated, and silently handles competing columns, thereby reducing the input size and avoiding false sufficiency caused by similar fields in the same document.

[0045] In one embodiment of the present invention, the compression termination condition is defined as whether there are coherent data cells in the candidate context. If no coherent data cells are found, the data is added in the following order: columns adjacent to the current search position within the same table that do not match the interference word set in the belonging header set; the previous clause of the current clause; and columns in the same header set that match the keyword set in the belonging header set. The calculation is then repeated. It will stop when the preset upper limit is reached.

[0046] S4. Input the compressed candidate context into the generation model to obtain candidate value text; search for a continuous string that matches the candidate value text in the compressed candidate context, and locate the position of the retrieved string as the value range of the candidate value text in the compressed candidate context.

[0047] The generative model is a language model that generates corresponding field value texts based on the input text. It can be a sequence-to-sequence model or a pre-trained language model. The compressed candidate context and the field binding specification of the current field are input into the generative model, which then outputs the candidate value texts. The generative model's role is limited to producing candidate value texts; it does not use the character positions, offsets, or citation numbers output by the model as the value range. The value range is determined by the next step of searching the compressed candidate context for a continuous string that is completely identical to the candidate value text. If no completely identical continuous string is found, the candidate value text is determined not to be traceable to the source document, and extraction is marked as failed.

[0048] like Figure 2 As shown, specifically, step S4 includes: A generative model is used to generate candidate value text in the compressed candidate context, without using the character position output by the generative model as the value range; Search for a continuous string that is completely consistent with the candidate value text in the compressed candidate context. Each time a match is found, the start and end positions of the match are recorded as a value interval. If no match is found, it is determined that the candidate value text cannot be traced back to the source document and the extraction is marked as failed. If one or more value intervals are found, proceed to step S5.

[0049] S5. Determine whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint. The evidence includes at least one of the following: a set of attribution headers, unit words, and negation trigger words. When the position constraint is satisfied, the field values ​​corresponding to the value range are extracted and normalized to obtain the structured extraction result; the positioning relationship of the value range relative to the set of the belonging table header is recorded and written to the cache; If the position constraint is not met, the extraction is marked as a failure and the failure mode is recorded. The failure mode is statistically analyzed according to the frequency of recurrence. When the frequency of recurrence reaches a preset threshold, a specification patch is generated and the field binding specification is written back.

[0050] Further, determining whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint specifically includes: When the value range originates from a table, the set of headers corresponding to the data cell containing the value range is read; when the value range originates from narrative text, the clause containing the value range is read; the positional relationship between the set of headers, unit words, and negative trigger words relative to the value range is checked, and the actual established positional relationship is selected according to the recorded order of positional constraints in the field binding specification, and the association determination result is calculated; wherein the positional relationship includes row, column, sentence, or character window: ; in, Indicates the range of values The association determination result is 1, which indicates that the value range passes the association determination, and 0, which indicates that the association determination fails. v To determine the character span for the value, For the coherence criteria of the table header, As a unit criterion, To exclude criteria; Will The range of values ​​is used as a candidate range for association; if there is only one candidate range for association, it is determined that the range of values ​​satisfies the positional constraint. When multiple candidate intervals exist that are linked, they are selected in the following order of priority: Value ranges derived from tables take precedence over value ranges derived from narrative text; table header constraints are ranked from highest to lowest; the number of constraints that are met is ranked from highest to lowest; character distance is ranked from lowest to highest; reading order is ranked from front to back; if the distinction cannot be made by priority, the ambiguity is marked as unsatisfactory and the position constraint is determined to be unmet. When there is only one associated candidate interval or one is selected according to priority, the value interval is determined to satisfy the positional constraint.

[0051] Among them, the header constraint is the depth of the innermost header node in the header set that matches the keyword set; the deeper the level, the higher the constraint. The constraint hit count is the number of bound positioning category items that match the value interval. The character distance is the minimum absolute value of the difference between the starting point of the value interval and the starting point of the evidence span that has been associated. The reading order is the order in which the value intervals are located in the source document.

[0052] Understandably, the process involves searching the compressed candidate context for consecutive strings that are completely identical to the candidate value text to be validated, and defining each such consecutive string as a value interval. If no such string is found, the candidate value text is determined to be irretrievable from the source document and extraction is marked as failed; if one or more value intervals are found, the process proceeds to position constraint determination. If the error message is 0 and the problem persists because the header set does not match the keyword set, or the compressed candidate context does not contain the column containing the correct data cell, then the remaining cells in the same table are searched using the value as the key. The relevant columns are then added to the compressed candidate context, and only the cells are recalculated. Do not return to step S3.

[0053] In this invention, the exclusion of narrative channels is calculated solely based on locking events within the same logical partition as the value range; the exclusion of table channels is calculated solely based on locking events within the scope of the header set to which they belong. The exclusion results for narrative channels and table channels are calculated separately, without multiplying the positional relationship of one channel by the positional relationship of the other. To avoid mistakenly applying the geometric conditions of one channel to another, position validity is established on a traceable value range. Measurement negations are restricted to the table header, and diagnostic negations are restricted to the value text and its logical partition, based on assertion subtypes, to prevent mistakenly applying negations from adjacent columns or clauses to the current value.

[0054] In one embodiment of the invention, negation trigger words are divided into enumerative and predicate types. The text following an enumerative negation trigger word is segmented into a sequence of noun items using commas, semicolons, and coordinating conjunctions, stopping at a period or adversative conjunction. The scope of a predicate-type negation trigger word is limited to the predicate and object of the clause containing the trigger word, and it does not cross a semicolon into the next clause. Each negation trigger word and its resulting segmented noun items constitute a locking event. Previous trigger words are included in the exclusion criteria according to the same segmentation rules as the negation trigger words.

[0055] When the assertion subtype is measurement-based, the negation trigger word is applied to the header node corresponding to the data cell containing the value range, rather than the numerical value itself. Its scope is limited to the set of headers to which it belongs. For example, in the auxiliary examination, it states "No postprandial blood glucose elevation observed, fasting blood glucose 5.6." The negation target is the header of "postprandial blood glucose." The set of headers to which the cell containing 5.6 belongs is "fasting blood glucose" and "blood glucose," which has no overlap with "postprandial." 1 is still selected, and 5.6 are not excluded. If it is written as "no increase in fasting blood glucose," the negative object falls into the head set of the attribution table. Take 0.

[0056] When asserting a subtype as diagnostic, the target of the negation trigger is determined to be the value text corresponding to the value range. The scope is limited to the logical partition containing the value range; nouns that cross logical partitions do not constitute exclusion. For example, if the past medical history is written as "deny diabetes," and the admission diagnosis is written as "type 2 diabetes," then diabetes appears as the value text in the admission diagnosis partition. The "deny" in the past medical history partition cannot cross partitions to exclude values. Nouns that cross logical partitions do not constitute exclusion. Past medical history triggers are included in the exclusion criteria according to the same segmentation rules as negation triggers.

[0057] In one embodiment of the present invention, when the field binding specification is recorded as a single-value field and the above conditions still cannot distinguish between them, no forced choice is made on the machine side; instead, it is marked as an ambiguity failure and follows the branch record failure mode in step S5 where the position constraint is not satisfied. Disambiguation only occurs in... If a candidate with a value of 1 is selected or declared to have failed, the association determination result of any candidate is not written back or modified.

[0058] The range of values ​​has been determined, but If the failure is due to the header set not matching the keyword set, or the correct column not being in the compressed candidate context, then the remaining data cells in the same table of the source document are retrieved using the value text already retrieved via string search as the key (instead of searching in the compressed candidate context). This means that the string retrieved in the compressed candidate context in step S4 may come from the narrative text or residual headers from adjacent columns, and does not guarantee a corresponding coherent column; therefore, it is allowed to use this string to return to the source table to search for other cells. The coherent columns after being added are included in the candidates, and only the data cell is recalculated. The process does not return to step S3 for recompression, nor does it introduce new semantic retrieval. If the extraction still fails after supplementation, it remains an extraction failure.

[0059] In one embodiment of the present invention, when the position constraint is satisfied in step S5, the specific steps include: When the evidence corresponding to the value range contains unit words, the unit conversion and format normalization of the field values ​​corresponding to the value range are performed based on the unit words to obtain normalized field values; when the evidence corresponding to the value range does not contain unit words, the field values ​​corresponding to the value range are directly used as normalized field values. The in-text time expression recorded in the field binding specification is read as a time window, and it is determined whether the normalized field value falls within the time window; the normalized field value that falls within the time window is used as the field value that passes the verification, and the value that does not fall within the time window is not entered into the structured processing; the in-text time expression is not written as an independent field in the structured extraction result. The validated field values, the value ranges, the hit header nodes and negative trigger words and their corresponding positions, whether they come from tables or narrative text, and the version number of the field binding specification are organized into a structured extraction result. Read the validated field values ​​and corresponding value ranges from the structured extraction results, extract the positioning relationship of the value range relative to the set of the belonging table header, as well as the table header node, unit word, and negative trigger word, and write it to the cache; when processing new source documents with similar layouts, narrow the search range of candidate data cells according to the positioning relationship in the cache; regardless of whether the cache is hit, the value range is repositioned and the position constraint is re-determined, and hitting the cache does not replace the position constraint determination.

[0060] Furthermore, when the position constraint in step S5 is not satisfied, specifically including: Count the failure modes that are not written into the structured extraction results. The failure modes include any of the following: string retrieval failure, irrelevant table header, exclusion criterion trigger, correct column not entering the candidate context, ambiguity failure, and time window not falling into the target area. When the count of any failure mode reaches a preset threshold, a specification patch is generated and the keyword set, interference word set, or negative trigger word set is written back.

[0061] Understandably, in-text time expressions are not written as independent metadata to avoid misjudging genuine narrative records without explicit timestamps as invalid. Candidate value texts that do not meet positional constraints are not included in unit conversion. Unit conversion is performed according to the conversion relationships in the specification or unit thesaurus. Among them, the matching results of missing keyword sets, hitting negative trigger words or past trigger words within the scope are considered in the association determination and are used for calculation. , Only after this... Only conclusions drawn based on the number of candidate items are considered as either position constraint satisfied or position constraint not satisfied.

[0062] Failure modes include any of the following: (1) String retrieval failed: No completely identical continuous string was found in the compressed candidate context for the candidate value text; (2) Irrelevant header: The header set does not match the keyword set, or it matches the interference word set. Relevance criterion for header: ; (3) Exclusion criterion trigger: If the negation trigger or the previous trigger hits within the scope of the assertion subtype, the exclusion criterion is triggered. ; (4) The correct column is not included in the candidate context: The value range has been located, but the column is not retained in the compressed candidate context. It still fails after being filled in by the value text. (5) Ambiguity failure: The association determination results of multiple value ranges are all 1, and they still cannot be distinguished according to priority, and the field binding specification is recorded as a single value field; (6) Time window not included: The value of the normalized field does not fall within the time window recorded in the field binding specification.

[0063] Similarity in layout means that the new source document and the source document used for writing to the cache have the same header hierarchy structure, and the relative row and column positions of the fields to be extracted relative to the header nodes remain unchanged. When narrowing the search scope, the rows, columns, or logical partitions where the candidate data cells are located are limited according to the positioning relationship in the cache, without changing the judgment rules of steps S3 and S4.

[0064] The field binding specification includes negative trigger words and previous trigger words (mandatory evidence type). The scope is defined by assertion subtype. Measurement type: Negation is applied to the header node, and its scope is limited to the set of header nodes to which it belongs; Diagnostic type: The negation effect is applied to the value text, and its scope is limited to the logical partition in which it is located.

[0065] Enumerated types are segmented into noun items after the trigger word by commas, semicolons, or coordinating conjunctions, stopping at a period or transition point; predicate types are limited to the predicate and object of the clause they belong to, and do not cross semicolons. Each trigger word and its segmented noun items constitute a locking event. Hit across logical partitions does not constitute exclusion. Previous trigger words use the same segmentation rules.

[0066] Hit within range, If this value range is excluded, it is considered an exclusion criterion triggered; if not found, .

[0067] Association determination If any value is 0, If the range of values ​​is not specified, then this range cannot be used as a field value.

[0068] Patches are generated based on failure modes: When the threshold is reached for irrelevant headers or correct columns not entering the candidate context, the header name or its synonym appearing in the current search is added to the keyword set for that field; when the threshold for ambiguity failure is reached, the header name corresponding to the competing field is added to the interference word set; when the threshold for exclusion criteria is reached and the same negative trigger word or previous trigger words repeatedly cause false exclusions, the word is removed from the mandatory evidence type or its scope is narrowed. After the patch is written back, the version number of the specification bound to the field is updated, and the updated specification is used for subsequent extractions. No patches are generated for keyword sets or interference word sets for string retrieval failures or when the time window does not fall within the specified range.

[0069] This invention caches successful location relationships to reduce the search cost of similar layouts, while writing back failure patterns that occur repeatedly and have not yet been covered by the specification to the specification, so that the specification is supplemented with the layout variations in actual processing.

[0070] This invention pre-generates a keyword set, a distractor word set, and positional constraints for each field to be extracted. It distinguishes and retains table columns based on hierarchical header indexes and retains narrative text by sentence index matching before extraction and position verification. Compared to the processing method of first compressing the candidate context as a whole and then directly accepting the model output, this invention moves the definition of the evidence range to before extraction and adds position verification after the value is produced. When the positional constraint is satisfied, the positioning relationship of the value range relative to the belonging header set is recorded and written to the cache. When it is not satisfied, the field binding specification is written back according to the failure mode. This allows the reduction of input size and the traceability and verifiability of the value to be taken into account at the same time. Overall, it is beneficial to reduce the resource consumption when processing long documents and reduce the mismatch between the value and the evidence.

[0071] like Figure 3 As shown, the present invention also provides a hierarchical evidence retrieval and context compression system based on field reduction, used to implement the above-described hierarchical evidence retrieval and context compression method based on field reduction, comprising: The field binding module is used to obtain the set of fields to be extracted from the source document, and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set, the interference word set, and the positional constraint between the evidence and the value corresponding to the current field. The interference word set is obtained by taking the union of the keyword sets of all fields in the set of fields to be extracted except the current field. The positional constraint is used to limit the relative positional relationship that the evidence and the value should satisfy. The index building module is used to parse the source document to obtain header nodes, data cells and narrative text, and build a hierarchical header index based on the ancestor header nodes on the list header path and row header path; each data cell in the hierarchical header index corresponds to a set of ancestor header nodes, which serve as the header set to which the data cell belongs; the narrative text is divided into clauses and the adjacency relationship between clauses is recorded to obtain the clause index; The retrieval and compression module is used to match the belonging header set with the keyword set and the interference word set respectively. For table columns in the belonging header set that match the keyword set but not the interference word set, their header nodes and values ​​are retained. For table columns in the belonging header set that match the interference word set, regardless of whether they also match the keyword set, only the header nodes are retained and the values ​​are removed. Table columns in the belonging header set that do not match either the keyword set or the interference word set are not retained. The narrative text is matched with the keyword set and the interference word set according to the sentence index, and the corresponding content is retained. The retained table part is merged with the narrative text part to obtain the compressed candidate context. The interval positioning module is used to input the compressed candidate context into the generation model to obtain candidate value text; to search for a continuous string that is consistent with the candidate value text in the compressed candidate context; and to locate the position of the searched string as the value interval of the candidate value text in the compressed candidate context. The specification generation module is used to determine whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint, wherein the evidence includes at least one of the following: a set of attribution headers, unit words, and negation trigger words. When the position constraint is satisfied, the field values ​​corresponding to the value range are extracted and normalized to obtain the structured extraction result; the positioning relationship of the value range relative to the set of the belonging table header is recorded and written to the cache; If the position constraint is not met, the extraction is marked as a failure and the failure mode is recorded. The failure mode is statistically analyzed according to the frequency of recurrence. When the frequency of recurrence reaches a preset threshold, a specification patch is generated and the field binding specification is written back.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hierarchical evidence retrieval and context compression method based on field specification, characterized in that, include: S1. Obtain the set of fields to be extracted from the source document, and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set, the interference word set, and the positional constraint between the evidence and the value corresponding to the current field. The interference word set is obtained by taking the union of the keyword sets of all fields in the set of fields to be extracted except the current field. The positional constraint is used to limit the relative positional relationship that the evidence and the value should satisfy. S2. Parse the source document to obtain header nodes, data cells, and narrative text. Construct a hierarchical header index based on ancestor header nodes on the list header path and row header path. Each data cell in the hierarchical header index corresponds to a set of ancestor header nodes, which serve as the header set to which the data cell belongs. Divide the narrative text into clauses and record the adjacency relationships between clauses to obtain a clause index. S3. Match the belonging header set with the keyword set and the interference word set respectively. For table columns in the belonging header set that match the keyword set but not the interference word set, retain their header nodes and values. For table columns in the belonging header set that match the interference word set, regardless of whether they also match the keyword set, only retain the header nodes and remove the values. Table columns in the belonging header set that do not match either the keyword set or the interference word set are not retained. Match the narrative text with the keyword set and the interference word set according to the sentence index, and retain the corresponding matched content. Merge the retained table portion with the narrative text portion to obtain the compressed candidate context. S4. Input the compressed candidate context into the generation model to obtain candidate value text; search for a continuous string that is consistent with the candidate value text in the compressed candidate context, and locate the position of the searched string as the value range of the candidate value text in the compressed candidate context. S5. Determine whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint. The evidence includes at least one of the following: a set of attribution headers, unit words, and negation trigger words. When the position constraint is satisfied, the field values ​​corresponding to the value range are extracted and normalized to obtain the structured extraction result; the positioning relationship of the value range relative to the set of the belonging table header is recorded and written to the cache; If the position constraint is not met, the extraction is marked as a failure and the failure mode is recorded. The failure mode is statistically analyzed according to the frequency of recurrence. When the frequency of recurrence reaches a preset threshold, a specification patch is generated and the field binding specification is written back.

2. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: The process of generating field binding specifications for each field in the set of fields to be extracted specifically includes: Generate a keyword set based on the name and synonyms of the table header corresponding to the current field; The set of keywords corresponding to the headers of all fields in the field set to be extracted, excluding the current field, is merged into the set of pre-defined explicit antonyms to obtain the set of interference words. The explicit antonyms are taken only from the antonyms of the headers already recorded in the field set to be extracted and are not introduced from external knowledge bases. Write the set of mandatory evidence types for the current field in the field binding specification. The mandatory evidence types are selected from the table header, unit words, negative trigger words, and past trigger words. Write a time window in the field binding specification, wherein the time window is determined by the in-text time expression; The binding location category set is written in the field binding specification. The binding location category is a non-empty subset of the row, column, sentence, character window, and mutually exclusive location. When the binding location category includes both table row and column categories and character window, table row and column categories take precedence over character window. The binding location category set serves as a positional constraint between evidence and value. Write assertion subtypes in the field binding specification. Assertion subtypes include measurement type and diagnostic type. The field value corresponding to the measurement type is the measurement value, and the field value corresponding to the diagnostic type is the noun text.

3. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 2, characterized in that: Each evidence type in the set of mandatory evidence types is written into a supplementary strategy, which includes a supplementary order and a maximum number of supplementary attempts. The order of insertion is as follows: the first priority is the column in the same table that is adjacent to the current search position and whose header set does not match the set of interfering words; the second priority is the clause before the current clause; the third priority is the column in the same header set whose header set matches the set of keywords. When evidence that meets the requirements is found according to the order of insertion, the priority search is stopped; the upper limit of the number of insertions is used to limit the maximum number of searches that can be performed on a single field during the insertion process, and the insertion is terminated after the upper limit of the number of insertions is reached.

4. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: Step S2 specifically includes: Identify table areas and narrative text areas in the source document; For the header cells in the table area, build a header path tree according to the inclusion relationship from the outermost to the innermost layer. The node sequence from the root node to the leaf node constitutes the table header path and the row header path, respectively. For each data cell, backtrack along the column and row directions on the header path tree, and form the ancestor header nodes on the backtracking path into a set of belonging headers; The narrative text area is segmented into clauses according to punctuation, and the adjacency relationship between clauses is recorded to obtain the clause index.

5. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 4, characterized in that: The backtracking along the column and row directions of each data cell in the table header path tree specifically includes: The innermost header node of the column containing the data cell is used as the column starting point, and the innermost header node of the row containing the data cell is used as the row starting point. Starting from the column, proceed along the column header path to the root node and take parent nodes layer by layer. Starting from the column, proceed along the row header path to the root node and take parent nodes layer by layer. Include the header nodes that are backtracked into the parent header set. Other child nodes under the same parent node that do not appear on the backtracking path are not included in the parent header set. When the header node on the backtracking path is a merged cell, the merged cell is included in the parent header set as the same ancestor node shared by multiple header paths or row header paths, and the data cells covered by the same merged cell point to the same ancestor header node.

6. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: Step S3 specifically includes: Calculate the coherence of data cells in the hierarchical header index: ； Where π represents the data cell in the hierarchical header index, and f represents the current field. For header coherence, For header exclusivity, For unit satisfiability, The relevance of data cell π to the current field f; Calculate coherence for each data cell in the hierarchical header index. ,Will The column containing the data cell with a value of 1 should retain its header node and value completely. The column containing cells with a value equal to 1 retains only the header nodes and removes the values; if all data cells... If none of the values ​​are equal to 1, then it is determined that there are no relevant data cells in the current field; If no relevant data cells exist, first fill in the table with columns adjacent to the current search position that do not match the interference word set in the header set; if no relevant data cells still exist after filling in, then fill in columns with the same header set that match the keyword set in the header set; in the narrative text, fill in the clause preceding the current clause, and recalculate each time it is filled in. Stop when the maximum number of replenishments is reached; Based on the clause index, clauses that match the keyword set and their adjacent clauses are retained, while other clauses are not retained. The retained narrative text is arranged after the table columns retained by the matching retrieval and incorporated into the compressed candidate context.

7. The hierarchical evidence retrieval and context compression method based on field specification as described in claim 1, characterized in that: Step S4 specifically includes: A generative model is used to generate candidate value text in the compressed candidate context, without using the character position output by the generative model as the value range; Search for a continuous string that is completely consistent with the candidate value text in the compressed candidate context. Each time a match is found, the start and end positions of the match are recorded as a value interval. If no match is found, it is determined that the candidate value text cannot be traced back to the source document and the extraction is marked as failed. If one or more value intervals are found, proceed to step S5.

8. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: The determination of whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint specifically includes: When the value range originates from a table, the set of headers corresponding to the data cell containing the value range is read; when the value range originates from narrative text, the clause containing the value range is read; the positional relationship between the set of headers, unit words, and negative trigger words relative to the value range is checked, and the actual established positional relationship is selected according to the recorded order of positional constraints in the field binding specification, and the association determination result is calculated; wherein the positional relationship includes row, column, sentence, or character window: ； in, Indicates the range of values The association determination result is 1, which indicates that the value range passes the association determination, and 0, which indicates that the association determination fails. v To determine the character span for the value, For the coherence criteria of the table header, As a unit criterion, To exclude criteria; Will The range of values ​​is used as a candidate range for association; if there is only one candidate range for association, it is determined that the range of values ​​satisfies the positional constraint. When multiple candidate intervals exist that are linked, they are selected in the following order of priority: Value ranges derived from tables take precedence over value ranges derived from narrative text; table header constraints are ranked from highest to lowest; the number of constraints that are met is ranked from highest to lowest; character distance is ranked from lowest to highest; reading order is ranked from front to back; if the distinction cannot be made by priority, the ambiguity is marked as unsatisfactory and the position constraint is determined to be unmet. When there is only one associated candidate interval or one is selected according to priority, the value interval is determined to satisfy the positional constraint.

9. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: When the position constraint is satisfied in step S5, it specifically includes: When the evidence corresponding to the value range contains unit words, the unit conversion and format normalization of the field values ​​corresponding to the value range are performed based on the unit words to obtain normalized field values; when the evidence corresponding to the value range does not contain unit words, the field values ​​corresponding to the value range are directly used as normalized field values. The in-text time expression recorded in the field binding specification is read as a time window, and it is determined whether the normalized field value falls within the time window; the normalized field value that falls within the time window is used as the field value that passes the verification, and the value that does not fall within the time window is not entered into the structured processing; the in-text time expression is not written as an independent field in the structured extraction result. The validated field values, the value ranges, the hit header nodes and negative trigger words and their corresponding positions, whether they come from tables or narrative text, and the version number of the field binding specification are organized into a structured extraction result. Read the validated field values ​​and corresponding value ranges from the structured extraction results, extract the positioning relationship of the value range relative to the set of the belonging table header, as well as the table header node, unit word, and negative trigger word, and write it to the cache; when processing new source documents with similar layouts, narrow the search range of candidate data cells according to the positioning relationship in the cache; regardless of whether the cache is hit, the value range is repositioned and the position constraint is re-determined, and hitting the cache does not replace the position constraint determination.

10. The hierarchical evidence retrieval and context compression method based on field reduction as described in claim 1, characterized in that: When the position constraint in step S5 is not satisfied, specifically including: Count the failure modes that are not written into the structured extraction results. The failure modes include any of the following: string retrieval failure, irrelevant table header, exclusion criterion trigger, correct column not entering the candidate context, ambiguity failure, and time window not falling into the target area. When the count of any failure mode reaches a preset threshold, a specification patch is generated and the keyword set, interference word set, or negative trigger word set is written back.

11. A hierarchical evidence retrieval and context compression system based on field specification, characterized in that, The system is used to implement the hierarchical evidence retrieval and context compression method based on field reduction as described in any one of claims 1-10, including: The field binding module is used to obtain the set of fields to be extracted from the source document, and generate a field binding specification for each field in the set of fields to be extracted. The field binding specification includes the keyword set, the interference word set, and the positional constraint between the evidence and the value corresponding to the current field. The interference word set is obtained by taking the union of the keyword sets of all fields in the set of fields to be extracted except the current field. The positional constraint is used to limit the relative positional relationship that the evidence and the value should satisfy. The index building module is used to parse the source document to obtain header nodes, data cells and narrative text, and build a hierarchical header index based on the ancestor header nodes on the list header path and row header path; each data cell in the hierarchical header index corresponds to a set of ancestor header nodes, which serve as the header set to which the data cell belongs; the narrative text is divided into clauses and the adjacency relationship between clauses is recorded to obtain the clause index; The retrieval and compression module is used to match the belonging header set with the keyword set and the interference word set respectively. For table columns in the belonging header set that match the keyword set but not the interference word set, their header nodes and values ​​are retained. For table columns in the belonging header set that match the interference word set, regardless of whether they also match the keyword set, only the header nodes are retained and the values ​​are removed. Table columns in the belonging header set that do not match either the keyword set or the interference word set are not retained. The narrative text is matched with the keyword set and the interference word set according to the sentence index, and the corresponding content is retained. The retained table part is merged with the narrative text part to obtain the compressed candidate context. The interval positioning module is used to input the compressed candidate context into the generation model to obtain candidate value text; to search for a continuous string that is consistent with the candidate value text in the compressed candidate context; and to locate the position of the searched string as the value interval of the candidate value text in the compressed candidate context. The specification generation module is used to determine whether the relative positional relationship between the value range and its corresponding evidence satisfies the positional constraint, wherein the evidence includes at least one of the following: a set of attribution headers, unit words, and negation trigger words. When the position constraint is satisfied, the field values ​​corresponding to the value range are extracted and normalized to obtain the structured extraction result; the positioning relationship of the value range relative to the set of the belonging table header is recorded and written to the cache; If the position constraint is not met, the extraction is marked as a failure and the failure mode is recorded. The failure mode is statistically analyzed according to the frequency of recurrence. When the frequency of recurrence reaches a preset threshold, a specification patch is generated and the field binding specification is written back.

Images