A large model-based text data governance method
By using a text data governance method based on a large model, the report text structure is automatically parsed and table identifiers are matched, which solves the problem of low efficiency in establishing table reference relationships in long reports and achieves efficient and accurate data governance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 山东省高级人民法院
- Filing Date
- 2025-11-14
- Publication Date
- 2026-06-09
AI Technical Summary
In existing technologies, the citation relationships between tables and text in long reports rely on manual establishment, which is inefficient and prone to errors, especially when the report is revised, which can easily lead to citation errors or loss, and the process is cumbersome when there is a lack of professional knowledge.
By employing a large-model-based text data governance approach, the report text structure is parsed, headings at all levels are extracted and table citation probabilities are obtained, summary descriptions are generated using the large model, table identifiers are automatically matched and inserted, and structural analysis and probability estimation are integrated to improve accuracy.
It enables the automatic and accurate insertion of table markers into the report body, improving data governance efficiency and accuracy, and is suitable for efficient processing of long reports.
Smart Images
Figure CN121503443B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a text data governance method based on a large model. Background Technology
[0002] In various academic reports and analytical reports, tables are key elements that carry core data and support conclusions. Traditionally, authors establish the relationship between textual arguments and tabular data by manually inserting table labels (such as Table 1, Table 2) into the text. However, this manual approach has the following problems:
[0003] Manual annotation is inefficient and prone to errors. Especially for lengthy reports, authors need to repeatedly switch between the main text and the tables at the end, manually verifying the correspondence between data and arguments—a tedious process. Furthermore, changes to table numbering or placement during report revisions can easily lead to citation errors or missing citations. Additionally, when managing report text data where tables lack a connection to the main text, manually re-establishing this connection requires specialized domain knowledge and is extremely time-consuming, also resulting in a tedious process and a high risk of citation errors or missing citations. Summary of the Invention
[0004] To address the aforementioned problems, the present invention aims to provide a text data governance method based on a large model.
[0005] According to the present invention, a text data governance method based on a large model is provided, characterized in that the method includes the following steps:
[0006] S100, parse the target report text and extract its structural information; the structural information of the target report text includes the headings at all levels of the main body of the target report text; the target report text includes a main body and a table part; the table part includes several tables.
[0007] S200, obtain the probability of each heading referencing a table in the body of the target report text.
[0008] S300: Extract the content of each table in the table section and use the large model to generate a summary description of each table.
[0009] S400, match each sentence corresponding to the candidate title in the main text with the summary description and data of the candidate table; the candidate title is the title of the main text that meets the target conditions, the target conditions include the probability of referencing the table being greater than or equal to a preset threshold; the candidate table is any table in the table section.
[0010] S500: If a sentence corresponding to a candidate title in the main text matches the summary description and key data of a candidate table, then the corresponding position of that sentence is inserted into the identifier of the candidate table.
[0011] Furthermore, the probability of each heading referencing a table in the body of the target report text includes:
[0012] S210, for any title in the main text, the title and its corresponding content summary are segmented into words, and the segmentation results are matched against a preset keyword list; the preset keyword list includes several keywords and the weight of each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the main text.
[0013] S220, obtain the first initial probability of the title reference table based on the weight of the successfully matched keywords and the number of times they appear in the title and the corresponding content summary.
[0014] S230, obtain the category of the title, and obtain the second initial probability of the title referencing the table according to the category of the title and the preset mapping relationship; the preset mapping relationship includes the mapping relationship between the category and the probability of referencing the table.
[0015] S240, obtain the probability of the title reference table based on the first initial probability of the title reference table and the second initial probability of the title reference table.
[0016] Furthermore, S220 includes:
[0017] S221, For any keyword that is successfully matched in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of times the keyword appears in the title and the content summary corresponding to the title is determined as the initial value of the keyword.
[0018] S222, the initial values of keywords that are successfully matched in the title and the corresponding content summary are summed to obtain the intermediate value of the title.
[0019] S223, adjust the median value of the title based on the length of the title and the content summary corresponding to the title to obtain the adjusted title.
[0020] S224, normalize the correction result of the title to obtain the first initial probability of the title referencing the table.
[0021] Furthermore, S240 includes:
[0022] S241, obtain the first corrected probability of the title reference table based on the first initial probability of the title reference table and the preset first weight; the preset first weight is greater than 0.
[0023] S242, obtain the second corrected probability of the title reference table based on the second initial probability of the title reference table and the preset second weight; the preset second weight is greater than 0.
[0024] S243, the sum of the first modified probability and the second modified probability is determined as the probability of the title referencing the table.
[0025] Furthermore, a predefined chapter type classifier is used to obtain the category of the title. The predefined chapter type classifier is used to classify the title into a predefined category. In the preset mapping relationship, at least two different predefined categories have different probabilities of corresponding reference tables.
[0026] Furthermore, the predefined categories include Introduction, Background, Methods, Results, Discussion, Conclusion, and Others.
[0027] Furthermore, matching each sentence corresponding to the candidate title in the main text with the summary description and data in the candidate table includes:
[0028] S410, determine whether the specified sentence matches the summary description of the candidate table. If they match, proceed to S420; otherwise, determine that the specified sentence does not match the summary and data of the candidate table. The specified sentence is any sentence corresponding to the candidate title in the main text.
[0029] S420, use the large model to reason and judge the data of the specified sentence and the candidate table, prompt the large model to judge whether the specified sentence is inferred based on the data of the candidate table, and output a binary judgment result of yes or no;
[0030] S430, if the output is yes, then it is determined that the specified sentence matches the summary description and key data of the candidate table; otherwise, it is determined that the specified sentence does not match the summary and data of the candidate table.
[0031] Furthermore, the method also includes: if the content of a page in the target report text contains a table identifier, then displaying preset information corresponding to the table identifier in the footer of that page.
[0032] Furthermore, the method also includes: if the amount of information to be displayed at the footer position of a certain page in the target report text is greater than a preset information amount threshold, then the preset information corresponding to each identifier to be displayed at that footer position is simplified.
[0033] Compared with the prior art, the present invention has at least the following beneficial effects:
[0034] This invention analyzes the target report text, extracts the headings at each level of the main body, and obtains the probability that each heading references a table. For headings with a table reference probability greater than or equal to a preset threshold, it determines whether each sentence in the main body matches the table's summary description and data. If a match is found, a table identifier is inserted at the corresponding position of the matching sentence. Thus, by integrating report structure analysis, probability estimation, and large-scale model semantic understanding, this invention enables the system to automatically and accurately insert table identifiers at appropriate positions in the report text, improving the efficiency and accuracy of data governance. Furthermore, this invention can quickly focus on text areas with a high probability of table references, avoiding unnecessary calculations and can be efficiently applied to processing long report texts, demonstrating high practical value. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 A flowchart illustrating a text data governance method based on a large model, provided in an embodiment of the present invention. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] According to this embodiment, as Figure 1 As shown, a text data governance method based on a large model is provided, the method including the following steps:
[0039] S100, parse the target report text and extract its structural information; the structural information of the target report text includes the headings at all levels of the main body of the target report text; the target report text includes a main body and a table part; the table part includes several tables.
[0040] In this embodiment, the target report text is the report text to be processed, including the main body and tables. The target report text has a hierarchical heading structure. By identifying the headings at each level in the main body, the target report text can be logically divided, facilitating independent analysis of the content corresponding to each heading. As a specific implementation, the table section is located at the end of the target report text, that is, after the main body.
[0041] Those skilled in the art will know that document parsing tools in the prior art can be used to parse the target report text and obtain the various levels of headings included in the target report text, which will not be elaborated here.
[0042] S200, obtain the probability of each heading referencing a table in the body of the target report text.
[0043] As one specific implementation, the probability of obtaining each heading reference table in the body of the target report text includes:
[0044] S210, for any title in the main text, the title and its corresponding content summary are segmented into words, and the segmentation results are matched against a preset keyword list; the preset keyword list includes several keywords and the weight of each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the main text.
[0045] In this embodiment, the content corresponding to any title in the main text is the content under that title in the main text. The content corresponding to any title can be summarized by a large model to obtain a content summary corresponding to that title. The content summary corresponding to that title can capture the core semantics of the content corresponding to that title in the main text.
[0046] In this embodiment, any title and its corresponding content summary can reflect the theme of the content corresponding to the title in the main text. By matching the words in the title and its corresponding content summary with a preset keyword list, it is possible to evaluate whether the content corresponding to the title should be included in the table.
[0047] In this embodiment, the preset keyword list is a pre-established and updatable list. It includes several keywords and their corresponding weights. Different keywords may have the same or different weights. The higher the weight of a keyword, the greater the probability that the content corresponding to that title will be included in the table when that keyword appears in the title or the content summary corresponding to the title. For example, the preset keyword list includes high-weight keywords, medium-weight keywords, and low-weight keywords. High-weight keywords include table, data, statistics, results, indicators, comparison, ranking, distribution, etc.; medium-weight keywords include analysis, comparison, classification, measurement, experiment, performance, etc.; low-weight keywords include background, introduction, overview, etc. Optionally, high-weight and medium-weight keywords are both greater than 0, and keywords with the same high weight can have different weights, as can keywords with the same medium weight; low-weight keywords can be less than 0, and keywords with the same low weight can have different weights. It should be noted that the keywords included in the preset keyword list and the weights corresponding to different keywords can be updated and adjusted.
[0048] Those skilled in the art will know that word segmentation tools in the prior art can be used to segment the title and the corresponding content summary, which will not be elaborated here.
[0049] S220, obtain the first initial probability of the title reference table based on the weight of the successfully matched keywords and the number of times they appear in the title and the corresponding content summary.
[0050] In this embodiment, the first initial probability of the title reference table is positively correlated with the probability of the title reference table. As a specific implementation, S220 includes:
[0051] S221, For any keyword that is successfully matched in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of times the keyword appears in the title and the content summary corresponding to the title is determined as the initial value of the keyword.
[0052] In one specific implementation, if a keyword has a weight of 0.5 and appears twice in the title and the corresponding content summary, then the initial value of the keyword is 0.5 × 2 = 1.
[0053] S222, the initial values of keywords that are successfully matched in the title and the corresponding content summary are summed to obtain the intermediate value of the title.
[0054] It should be noted that if the median value of a title is less than 0, then 0 will be used as the median value of that title.
[0055] S223, adjust the median value of the title based on the length of the title and the content summary corresponding to the title to obtain the adjusted title.
[0056] In this embodiment, adjusting the median value of the title based on the length of the title and its corresponding content summary is to address the issue that longer titles (i.e., those containing more words) tend to have larger median values. As an optional implementation, the adjusted value for a title is c, where c = a / lg(b+1), where a is the median value of the title, b is the length of the title and its corresponding content summary, and lg is a logarithmic function to base 10. This reduces the impact of long texts on the median value without causing over-adjustment.
[0057] S224, normalize the correction result of the title to obtain the first initial probability of the title referencing the table.
[0058] Those skilled in the art will recognize that the normalization process is prior art and will not be described in detail here. For example, the Sigmoid function or Min-Max scaling is used to map the corrected result to the [0,1] interval.
[0059] Based on S221-S224, the probability of introducing the content corresponding to the title into a table can be estimated based on the title and the content corresponding to the title.
[0060] S230, obtain the category of the title, and obtain the second initial probability of the title referencing the table according to the category of the title and the preset mapping relationship; the preset mapping relationship includes the mapping relationship between the category and the probability of referencing the table.
[0061] In this embodiment, the probability of a title referencing a table is evaluated (i.e., the second initial probability) by introducing prior knowledge of the category of the title and the probability of referencing a table.
[0062] As one specific implementation, a predefined chapter type classifier is used to obtain the category of the title. This predefined chapter type classifier is used to classify the title into a predefined category, and in the preset mapping relationship, at least two different predefined categories correspond to reference tables with different probabilities. As another specific implementation, the predefined chapter type classifier classifies the title into a predefined category based on rules (such as title keyword matching) or machine learning models (such as SVM or BERT classifiers).
[0063] In this embodiment, the preset mapping relationship is a pre-established mapping relationship between categories and the probability of referencing tables (which can be established based on domain statistical knowledge). The probability of referencing tables corresponding to different predefined categories may be the same or different. As a specific implementation, the predefined categories include introduction, background, method, result, discussion, conclusion, and others. For example, when the category is result, the probability of referencing a table is 0.9; when the category is introduction, the probability of referencing a table is 0.1; and when the category is other, the probability of referencing a table is 0.5.
[0064] S240, obtain the probability of the title reference table based on the first initial probability of the title reference table and the second initial probability of the title reference table.
[0065] In one specific implementation, S240 includes:
[0066] S241, obtain the first corrected probability of the title reference table based on the first initial probability of the title reference table and the preset first weight; the preset first weight is greater than 0.
[0067] S242, obtain the second corrected probability of the title reference table based on the second initial probability of the title reference table and the preset second weight; the preset second weight is greater than 0.
[0068] In one specific implementation, the first weight and the second weight are empirical values, and the sum of the first weight and the second weight is 1. It should be understood that when the first weight is relatively large, it means that the first initial probability is referenced more; when the second weight is relatively large, it means that the second initial probability is referenced more.
[0069] S243, the sum of the first modified probability and the second modified probability is determined as the probability of the title referencing the table.
[0070] Based on S241-S243, by fusing text features (i.e., the first initial probability) and structural features (i.e., the second initial probability) to obtain the final probability, a more comprehensive probability assessment can be achieved, the bias of single feature assessment can be reduced, and the accuracy of the obtained probability can be improved.
[0071] S300: Extract the content of each table in the table section and use the large model to generate a summary description of each table.
[0072] In this embodiment, the large model is a large language model with powerful natural language understanding and generation capabilities; for example, the GPT series. In this embodiment, the large model is used to abstract and summarize the data in the table, resulting in a summary description of the table.
[0073] As a specific implementation method, an OCR tool is used to extract tabular data, and the extracted tabular data is input into a large model. The large model is prompted to summarize the trends and key data of the table and output a summary description of the table.
[0074] S400, match each sentence corresponding to the candidate title in the main text with the summary description and data of the candidate table; the candidate title is the title of the main text that meets the target conditions, the target conditions include the probability of referencing the table being greater than or equal to a preset threshold; the candidate table is any table in the table section.
[0075] As a specific implementation, if a heading in the main text does not meet the target conditions, it is determined that the content corresponding to that heading has a low probability of referencing a table, and each sentence corresponding to that heading is not matched with the summary description and data of the candidate table. Optionally, the preset threshold is an empirical value, such as a preset threshold of 0.6 or 0.7.
[0076] As a specific implementation method, matching each sentence corresponding to the candidate title in the main text with the summary description and data in the candidate table includes:
[0077] S410, determine whether the specified sentence matches the summary description of the candidate table. If they match, proceed to S420; otherwise, determine that the specified sentence does not match the summary and data of the candidate table. The specified sentence is any sentence corresponding to the candidate title in the main text.
[0078] As a specific implementation, the process of determining whether a specified sentence matches the summary description of the candidate table includes: using lightweight similarity calculation (such as word overlap rate) or vector similarity; if the similarity is greater than or equal to a preset similarity threshold (e.g., 0.5 or 0.6), it is determined to be a match; otherwise, it is determined to be a mismatch. This allows for rapid determination of whether a specified sentence matches the candidate table, and quick filtering of obviously irrelevant sentences.
[0079] S420, use the large model to infer and judge the data of the specified sentence and the candidate table, prompt the large model to judge whether the specified sentence is inferred based on the data of the candidate table, and output a binary judgment result of yes or no.
[0080] As one specific implementation, inputting a sentence and table data prompts the large model to make a binary judgment; optionally, the prompt words include: whether the sentence is inferred based on the table data, and the answer is yes or no.
[0081] S430, if the output is yes, then it is determined that the specified sentence matches the summary description and key data of the candidate table; otherwise, it is determined that the specified sentence does not match the summary and data of the candidate table.
[0082] Based on S410-S430, a quick judgment is made on whether the sentence and the table are related. Only when they are related is the large model started to perform deep reasoning on whether the sentence is based on the data obtained from the table. This can improve processing efficiency while ensuring the final accuracy.
[0083] S500: If a sentence corresponding to a candidate title in the main text matches the summary description and key data of a candidate table, then the corresponding position of that sentence is inserted into the identifier of the candidate table.
[0084] As one specific implementation, table identifiers can be inserted at the end of sentences or other locations. This establishes visual references, increasing the readability of the target report text and the traceability of the data.
[0085] In one specific implementation, the method further includes: if a page in the target report text contains a table identifier, then displaying preset information corresponding to the inserted table identifier in the footer of that page. In one specific implementation, the preset information corresponding to the table identifier includes the table number and a description of the table (e.g., the table title or key values). Therefore, by displaying information about the referenced table in the footer, users can be provided with quick reference, improving the user experience.
[0086] As a specific implementation, the method further includes: if the amount of information to be displayed at the footer position of a page in the target report text exceeds a preset information amount threshold, then simplifying the preset information corresponding to each identifier to be displayed at that footer position. Optionally, a large model is used for simplification. Optionally, the preset information amount threshold is an empirical value, such as the maximum amount of information that can be displayed at the footer position. Therefore, when there is a lot of information to be displayed in the footer, the information can be simplified to avoid information congestion and affect normal display.
[0087] This embodiment parses the target report text, extracts the headings at all levels of the main body, and obtains the probability that each heading references a table. For headings with a table reference probability greater than or equal to a preset threshold, it determines whether each sentence in the main body matches the table's summary description and data. If a match is found, the table's identifier is inserted at the corresponding position of the matching sentence. Therefore, by integrating report structure analysis, probability estimation, and large-scale model semantic understanding, this embodiment enables the system to automatically and accurately insert table identifiers at appropriate positions in the report text, improving the efficiency and accuracy of data governance. Furthermore, this embodiment can quickly focus on text areas with a high probability of table references, avoiding unnecessary calculations and can be efficiently applied to processing long report texts, demonstrating high practical value.
[0088] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims
1. A large model-based text data governance method, characterized by, The method includes the following steps: S100, parse the target report text and extract its structural information; the structural information of the target report text includes the headings at all levels of the main body of the target report text; the target report text includes a main body and a table section; the table section includes several tables; S200, obtain the probability that each heading in the body of the target report text references a table; S300: Extract the content of each table in the table section and use the large model to generate a summary description of each table; S400, Match each sentence corresponding to the candidate title in the main text with the summary description and data of the candidate table; the candidate title is the title in the main text that meets the target conditions, the target conditions include the probability of referencing the table being greater than or equal to a preset threshold; the candidate table is any table in the table section; S500, if a sentence corresponding to a candidate title in the main text matches the summary description and key data of a candidate table, then insert the corresponding position of that sentence into the candidate table's identifier. The probability of each heading referencing a table in the body of the target report text includes: S210, for any title in the main text, the title and its corresponding content summary are segmented into words, and the segmentation results are matched against a preset keyword list; the preset keyword list includes several keywords and the weight of each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the main text. S220, Obtain the first initial probability of the title citation table based on the weight of the successfully matched keywords and the number of times they appear in the title and the content summary corresponding to the title; S230, obtain the category of the title, and obtain the second initial probability of the title referencing a table based on the category of the title and a preset mapping relationship; the preset mapping relationship includes the mapping relationship between the category and the probability of referencing a table; S240, obtain the probability of the title reference table based on the first initial probability of the title reference table and the second initial probability of the title reference table.
2. The large model-based text data governance method of claim 1, wherein, S220 includes: S221, For any keyword that is successfully matched in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of times the keyword appears in the title and the content summary corresponding to the title is determined as the initial value of the keyword; S222, the initial values of the keywords that are successfully matched in the title and the corresponding content summary are summed to obtain the intermediate value of the title; S223, adjust the median value of the title based on the length of the title and the content summary corresponding to the title to obtain the adjusted title result; S224, normalize the correction result of the title to obtain the first initial probability of the title referencing the table.
3. The large model-based text data governance method of claim 1, wherein, S240 includes: S241, Obtain the first corrected probability of the title reference table based on the first initial probability of the title reference table and the preset first weight; the preset first weight is greater than 0; S242, obtain the second corrected probability of the title reference table based on the second initial probability of the title reference table and the preset second weight; the preset second weight is greater than 0; S243, the sum of the first modified probability and the second modified probability is determined as the probability of the title referencing the table.
4. The large model-based text data governance method of claim 1, wherein, The category of the title is obtained using a predefined chapter type classifier, which is used to classify the title into a predefined category. In the preset mapping relationship, at least two different predefined categories have different probabilities of corresponding reference tables.
5. The text data governance method based on a large model according to claim 4, characterized in that, The predefined categories include Introduction, Background, Methods, Results, Discussion, Conclusion, and Other.
6. The text data governance method based on a large model according to claim 1, characterized in that, Matching each sentence corresponding to a candidate title in the main text with the summary description and data in the candidate table includes: S410, determine whether the specified sentence matches the summary description of the candidate table. If they match, proceed to S420; otherwise, determine that the specified sentence does not match the summary and data of the candidate table. The specified sentence is any sentence corresponding to the candidate title in the main text. S420, use the large model to reason and judge the data of the specified sentence and the candidate table, prompt the large model to judge whether the specified sentence is inferred based on the data of the candidate table, and output a binary judgment result of yes or no; S430, if the output is yes, then it is determined that the specified sentence matches the summary description and key data of the candidate table; otherwise, it is determined that the specified sentence does not match the summary and data of the candidate table.
7. The text data governance method based on a large model according to claim 1, characterized in that, The method further includes: if the content of a page in the target report text contains a table identifier, then displaying preset information corresponding to the table identifier in the footer of that page.
8. The text data governance method based on a large model according to claim 7, characterized in that, The method further includes: if the amount of information to be displayed at the footer position of a page in the target report text is greater than a preset information threshold, then the preset information corresponding to each identifier to be displayed at that footer position is simplified.