Text data governance method based on large model

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a text data governance method based on a large model. The method comprises the steps that a target report text is analyzed, and structural information of the target report text is extracted; obtaining the probability of each title reference table of the text part of the target report text; extracting the content of each table in the table part, and generating abstract description of each table by using a large model; matching each sentence corresponding to the candidate title in the text part with the abstract description and data of the candidate table; and if a certain sentence corresponding to the candidate title in the text part is matched with the abstract description and the key data of the candidate table, inserting the corresponding position of the sentence into the identifier of the candidate table. The table identification can be automatically inserted into the report text, and the insertion efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a text data governance method based on a large model. BACKGROUND

[0002] In various academic reports or analysis reports, tables are key elements that carry core data and support argument conclusions. Traditionally, authors establish the reference relationship between text argumentation and table data by manually inserting table identifiers (such as Table 1, Table 2) in the text. However, this manual approach has the following problems: Manual annotation is inefficient and prone to errors. Especially for long reports, authors need to repeatedly switch between the text and the table at the end of the text, manually check the correspondence between the data and the argumentation, which is tedious; and when modifying the report, changes in table numbers or positions can easily lead to reference errors or loss of references. In addition, when governing report text data that lacks a correlation between tables and text, manually re-establishing this correlation requires professional knowledge and is time-consuming, and similarly, the process is tedious and prone to reference errors or loss of references. SUMMARY

[0003] To solve the above problems, the present application aims to provide a text data governance method based on a large model.

[0004] According to the present application, a text data governance method based on a large model is provided, characterized in that the method comprises the following steps: S100, parsing a target report text to extract the structure information of the target report text; the structure information of the target report text includes the headings of each level of the text part of the target report text; the target report text includes a text part and a table part; the table part includes a plurality of tables.

[0005] S200, obtaining the probability of each heading of the text part of the target report text referencing a table.

[0006] S300, extracting the content of each table in the table part and using a large model to generate a summary description of each table.

[0007] S400, matching each sentence corresponding to a candidate heading in the text part with the summary description and data of a candidate table; the candidate heading is a heading of the text part that meets a target condition, the target condition including a probability of referencing a table greater than or equal to a preset threshold; the candidate table is any table in the table part.

[0008] S500, if a certain sentence corresponding to a candidate heading in the text part matches the summary description and key data of a candidate table, inserting the corresponding position of the sentence into the identifier of the candidate table.

[0009] Further, the acquiring the probability that each title of the body part of the target report text references the table comprises: In S210, for any title of the body part, the title and the content summary corresponding to the title are segmented, and the segmentation result is matched in a preset keyword list; the preset keyword list includes a plurality of keywords and a weight corresponding to each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the body part.

[0010] In S220, a first initial probability that the title references the table is acquired according to the weight of the matched keyword and the number of occurrences in the title and the content summary corresponding to the title.

[0011] In S230, the category of the title is acquired, and a second initial probability that the title references the table is acquired according to the category of the title and a preset mapping relationship; the preset mapping relationship includes a mapping relationship between the category and the probability of referencing the table.

[0012] In S240, the probability that the title references the table is acquired according to the first initial probability that the title references the table and the second initial probability that the title references the table.

[0013] Further, S220 comprises: In S221, for any matched keyword in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of occurrences of the keyword in the title and the content summary corresponding to the title is determined as the initial value of the keyword.

[0014] In S222, the initial values of the keywords matched in the title and the content summary corresponding to the title are accumulated to obtain an intermediate value of the title.

[0015] In S223, the intermediate value of the title is corrected according to the length of the title and the content summary corresponding to the title to obtain a correction result of the title.

[0016] In S224, the correction result of the title is normalized to obtain the first initial probability that the title references the table.

[0017] Further, S240 comprises: In S241, a first correction probability that the title references the table is acquired according to the first initial probability that the title references the table and a preset first weight; the preset first weight is greater than 0.

[0018] In S242, a second correction probability that the title references the table is acquired according to the second initial probability that the title references the table and a preset second weight; the preset second weight is greater than 0.

[0019] S243, determining the sum of the first correction probability and the second correction probability as the probability of the table of references of the title.

[0020] Further, a category of the title is obtained using a predefined chapter type classifier, the predefined chapter type classifier is used to divide the title into predefined categories, and the probability of the table of references corresponding to at least two different predefined category pairs in the preset mapping relationship is different.

[0021] Further, the predefined categories include introduction, background, method, result, discussion, conclusion, and others.

[0022] Further, matching each sentence corresponding to the candidate title in the text part with the abstract description and data of the candidate table includes: S410, determining whether the specified sentence and the abstract description of the candidate table match, if yes, entering S420; otherwise, determining that the specified sentence and the abstract and data of the candidate table do not match; the specified sentence is any sentence corresponding to the candidate title in the text part; S420, using a large model to infer and determine the specified sentence and the data of the candidate table, prompting the large model to determine whether the specified sentence is inferred based on the data of the candidate table, and outputting a binary determination result of yes or no; S430, if the output is yes, determining that the specified sentence matches the abstract description and the key data of the candidate table; otherwise, determining that the specified sentence and the abstract and data of the candidate table do not match.

[0023] Further, the method further includes: if the content included in a page in the target report text is inserted into a table, displaying the preset information corresponding to the identifier of the inserted table at the footer position of the page.

[0024] Further, the method further includes: if the amount of information to be displayed at the footer position of a page in the target report text is greater than a preset information amount threshold, simplifying the preset information corresponding to each identifier to be displayed at the footer position.

[0025] Compared with the prior art, the present application has at least the following beneficial effects: The present application extracts the titles of each level of the body part of the target report text by analyzing the target report text, and obtains the probability of each title referencing a table; and for the title whose probability of referencing a table is greater than or equal to a preset threshold, it is judged whether each sentence in the body part corresponding to the title matches the summary description and data of the table; if matched, the identifier of the table is inserted at the corresponding position of the matched sentence. Thus, by fusing report structure analysis, probability estimation and large model semantic understanding, the system can automatically and relatively accurately insert the table identifier at the corresponding position of the report body, improving the efficiency and accuracy of data governance. Moreover, the present application can quickly focus on the text area with high probability of referencing a table, avoiding unnecessary calculation, and can be efficiently applied to the processing scene of long report texts, having high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0027] Figure 1 The flowchart of the text data governance method based on a large model provided by the embodiments of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0029] According to the present embodiment, as shown in Figure 1 , a text data governance method based on a large model is provided, which comprises the following steps: S100, analyzing a target report text and extracting the structure information of the target report text; the structure information of the target report text includes the titles of each level of the body part of the target report text; the target report text includes a body part and a table part; the table part includes several tables.

[0030] In this embodiment, the target report text is a report text to be processed, including a body and a table part. The target report text has a hierarchical title structure. By identifying the titles at each level of the body part, the logical division of the target report text can be achieved, which facilitates independent analysis of the content corresponding to each title. As a specific implementation, the table part is located at the end of the target report text, i.e., after the body.

[0031] Those skilled in the art know that the document parsing tool in the prior art can be used to parse the target report text, and then obtain the titles at each level included in the target report text, which will not be described here.

[0032] S200, obtaining the probability that each title of the body part of the target report text references a table.

[0033] As a specific implementation, obtaining the probability that each title of the body part of the target report text references a table includes: S210, for any title of the body part, performing word segmentation on the title and the content summary corresponding to the title, and matching the word segmentation results in a preset keyword list; the preset keyword list includes a plurality of keywords and a weight corresponding to each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the body part.

[0034] In this embodiment, the content corresponding to any title in the body part, i.e., the content under the title in the body part, can be summarized by a large model to obtain a content summary corresponding to the title; the content summary corresponding to the title can capture the core semantics of the content corresponding to the title in the body part.

[0035] In this embodiment, any title and the content summary corresponding to the title can reflect the theme of the content corresponding to the title in the body part. By matching the words in the title and the content summary corresponding to the title in the preset keyword list, the evaluation of whether the content corresponding to the title introduces a table can be achieved.

[0036] In this embodiment, the preset keyword list is a pre-established and updatable list. It includes several keywords and their corresponding weights. Different keywords may have the same or different weights. The higher the weight of a keyword, the greater the probability that the content corresponding to that title will be included in the table when that keyword appears in the title or the content summary corresponding to the title. For example, the preset keyword list includes high-weight keywords, medium-weight keywords, and low-weight keywords. High-weight keywords include table, data, statistics, results, indicators, comparison, ranking, distribution, etc.; medium-weight keywords include analysis, comparison, classification, measurement, experiment, performance, etc.; low-weight keywords include background, introduction, overview, etc. Optionally, high-weight and medium-weight keywords are both greater than 0, and keywords with the same high weight can have different weights, as can keywords with the same medium weight; low-weight keywords can be less than 0, and keywords with the same low weight can have different weights. It should be noted that the keywords included in the preset keyword list and the weights corresponding to different keywords can be updated and adjusted.

[0037] Those skilled in the art will know that word segmentation tools in the prior art can be used to segment the title and the corresponding content summary, which will not be elaborated here.

[0038] S220, obtain the first initial probability of the title reference table based on the weight of the successfully matched keywords and the number of times they appear in the title and the corresponding content summary.

[0039] In this embodiment, the first initial probability of the title reference table is positively correlated with the probability of the title reference table. As a specific implementation, S220 includes: S221, For any keyword that is successfully matched in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of times the keyword appears in the title and the content summary corresponding to the title is determined as the initial value of the keyword.

[0040] In one specific implementation, if a keyword has a weight of 0.5 and appears twice in the title and the corresponding content summary, then the initial value of the keyword is 0.5 × 2 = 1.

[0041] S222, the initial values ​​of keywords that are successfully matched in the title and the corresponding content summary are summed to obtain the intermediate value of the title.

[0042] It should be noted that if the median value of a title is less than 0, then 0 will be used as the median value of that title.

[0043] S223, adjust the median value of the title based on the length of the title and the content summary corresponding to the title to obtain the adjusted title.

[0044] In this embodiment, adjusting the median value of the title based on the length of the title and its corresponding content summary is to address the issue that longer titles (i.e., those containing more words) tend to have larger median values. As an optional implementation, the adjusted value for a title is c, where c = a / lg(b+1), where a is the median value of the title, b is the length of the title and its corresponding content summary, and lg is a logarithmic function to base 10. This reduces the impact of long texts on the median value without causing over-adjustment.

[0045] S224, normalize the correction result of the title to obtain the first initial probability of the title referencing the table.

[0046] Those skilled in the art will recognize that the normalization process is prior art and will not be described in detail here. For example, the Sigmoid function or Min-Max scaling is used to map the corrected result to the [0,1] interval.

[0047] Based on S221-S224, the probability of introducing the content corresponding to the title into a table can be estimated based on the title and the content corresponding to the title.

[0048] S230, obtain the category of the title, and obtain the second initial probability of the title referencing the table according to the category of the title and the preset mapping relationship; the preset mapping relationship includes the mapping relationship between the category and the probability of referencing the table.

[0049] In this embodiment, the probability of a title referencing a table is evaluated (i.e., the second initial probability) by introducing prior knowledge of the category of the title and the probability of referencing a table.

[0050] As one specific implementation, a predefined chapter type classifier is used to obtain the category of the title. This predefined chapter type classifier is used to classify the title into a predefined category, and in the preset mapping relationship, at least two different predefined categories correspond to reference tables with different probabilities. As another specific implementation, the predefined chapter type classifier classifies the title into a predefined category based on rules (such as title keyword matching) or machine learning models (such as SVM or BERT classifiers).

[0051] In this embodiment, the preset mapping relationship is a pre-established mapping relationship between categories and the probability of referencing tables (which can be established based on domain statistical knowledge). The probability of referencing tables corresponding to different predefined categories may be the same or different. As a specific implementation, the predefined categories include introduction, background, method, result, discussion, conclusion, and others. For example, when the category is result, the probability of referencing a table is 0.9; when the category is introduction, the probability of referencing a table is 0.1; and when the category is other, the probability of referencing a table is 0.5.

[0052] S240, obtain the probability of the title reference table based on the first initial probability of the title reference table and the second initial probability of the title reference table.

[0053] In one specific implementation, S240 includes: S241, obtain the first corrected probability of the title reference table based on the first initial probability of the title reference table and the preset first weight; the preset first weight is greater than 0.

[0054] S242, obtain the second corrected probability of the title reference table based on the second initial probability of the title reference table and the preset second weight; the preset second weight is greater than 0.

[0055] In one specific implementation, the first weight and the second weight are empirical values, and the sum of the first weight and the second weight is 1. It should be understood that when the first weight is relatively large, it means that the first initial probability is referenced more; when the second weight is relatively large, it means that the second initial probability is referenced more.

[0056] S243, the sum of the first modified probability and the second modified probability is determined as the probability of the title referencing the table.

[0057] Based on S241-S243, by fusing text features (i.e., the first initial probability) and structural features (i.e., the second initial probability) to obtain the final probability, a more comprehensive probability assessment can be achieved, the bias of single feature assessment can be reduced, and the accuracy of the obtained probability can be improved.

[0058] S300: Extract the content of each table in the table section and use the large model to generate a summary description of each table.

[0059] In this embodiment, the large model is a large language model with powerful natural language understanding and generation capabilities; for example, the GPT series. In this embodiment, the large model is used to abstract and summarize the data in the table, resulting in a summary description of the table.

[0060] As a specific implementation method, an OCR tool is used to extract tabular data, and the extracted tabular data is input into a large model. The large model is prompted to summarize the trends and key data of the table and output a summary description of the table.

[0061] S400, match each sentence corresponding to the candidate title in the main text with the summary description and data of the candidate table; the candidate title is the title of the main text that meets the target conditions, the target conditions include the probability of referencing the table being greater than or equal to a preset threshold; the candidate table is any table in the table section.

[0062] As a specific implementation, if a heading in the main text does not meet the target conditions, it is determined that the content corresponding to that heading has a low probability of referencing a table, and each sentence corresponding to that heading is not matched with the summary description and data of the candidate table. Optionally, the preset threshold is an empirical value, such as a preset threshold of 0.6 or 0.7.

[0063] As a specific implementation method, matching each sentence corresponding to the candidate title in the main text with the summary description and data in the candidate table includes: S410, determine whether the specified sentence matches the summary description of the candidate table. If they match, proceed to S420; otherwise, determine that the specified sentence does not match the summary and data of the candidate table. The specified sentence is any sentence corresponding to the candidate title in the main text.

[0064] As a specific implementation, the process of determining whether a specified sentence matches the summary description of the candidate table includes: using lightweight similarity calculation (such as word overlap rate) or vector similarity; if the similarity is greater than or equal to a preset similarity threshold (e.g., 0.5 or 0.6), it is determined to be a match; otherwise, it is determined to be a mismatch. This allows for rapid determination of whether a specified sentence matches the candidate table, and quick filtering of obviously irrelevant sentences.

[0065] S420, use the large model to infer and judge the data of the specified sentence and the candidate table, prompt the large model to judge whether the specified sentence is inferred based on the data of the candidate table, and output a binary judgment result of yes or no.

[0066] As one specific implementation, inputting a sentence and table data prompts the large model to make a binary judgment; optionally, the prompt words include: whether the sentence is inferred based on the table data, and the answer is yes or no.

[0067] S430, if the output is yes, then it is determined that the specified sentence matches the summary description and key data of the candidate table; otherwise, it is determined that the specified sentence does not match the summary and data of the candidate table.

[0068] Based on S410-S430, a quick judgment is made on whether the sentence and the table are related. Only when they are related is the large model started to perform deep reasoning on whether the sentence is based on the data obtained from the table. This can improve processing efficiency while ensuring the final accuracy.

[0069] S500: If a sentence corresponding to a candidate title in the main text matches the summary description and key data of a candidate table, then the corresponding position of that sentence is inserted into the identifier of the candidate table.

[0070] As one specific implementation, table identifiers can be inserted at the end of sentences or other locations. This establishes visual references, increasing the readability of the target report text and the traceability of the data.

[0071] In one specific implementation, the method further includes: if a page in the target report text contains a table identifier, then displaying preset information corresponding to the inserted table identifier in the footer of that page. In one specific implementation, the preset information corresponding to the table identifier includes the table number and a description of the table (e.g., the table title or key values). Therefore, by displaying information about the referenced table in the footer, users can be provided with quick reference, improving the user experience.

[0072] As a specific implementation, the method further includes: if the amount of information to be displayed at the footer position of a page in the target report text exceeds a preset information amount threshold, then simplifying the preset information corresponding to each identifier to be displayed at that footer position. Optionally, a large model is used for simplification. Optionally, the preset information amount threshold is an empirical value, such as the maximum amount of information that can be displayed at the footer position. Therefore, when there is a lot of information to be displayed in the footer, the information can be simplified to avoid information congestion and affect normal display.

[0073] This embodiment parses the target report text, extracts the headings at all levels of the main body, and obtains the probability that each heading references a table. For headings with a table reference probability greater than or equal to a preset threshold, it determines whether each sentence in the main body matches the table's summary description and data. If a match is found, the table's identifier is inserted at the corresponding position of the matching sentence. Therefore, by integrating report structure analysis, probability estimation, and large-scale model semantic understanding, this embodiment enables the system to automatically and accurately insert table identifiers at appropriate positions in the report text, improving the efficiency and accuracy of data governance. Furthermore, this embodiment can quickly focus on text areas with a high probability of table references, avoiding unnecessary calculations and can be efficiently applied to processing long report texts, demonstrating high practical value.

[0074] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A text data governance method based on a large model, characterized in that, The method includes the following steps: S100, parse the target report text and extract its structural information; the structural information of the target report text includes the headings at all levels of the main body of the target report text; the target report text includes a main body and a table section; the table section includes several tables; S200, obtain the probability that each heading in the body of the target report text references a table; S300: Extract the content of each table in the table section and use the large model to generate a summary description of each table; S400, Match each sentence corresponding to the candidate title in the main text with the summary description and data of the candidate table; the candidate title is the title in the main text that meets the target conditions, the target conditions include the probability of referencing the table being greater than or equal to a preset threshold; the candidate table is any table in the table section; S500: If a sentence corresponding to a candidate title in the main text matches the summary description and key data of a candidate table, then the corresponding position of that sentence is inserted into the identifier of the candidate table.

2. The text data governance method based on a large model according to claim 1, characterized in that, The probability of each heading referencing a table in the body of the target report text includes: S210, for any title in the main text, the title and its corresponding content summary are segmented into words, and the segmentation results are matched against a preset keyword list; the preset keyword list includes several keywords and the weight of each keyword; the content summary corresponding to the title is obtained by summarizing the content corresponding to the title in the main text. S220, Obtain the first initial probability of the title citation table based on the weight of the successfully matched keywords and the number of times they appear in the title and the content summary corresponding to the title; S230, obtain the category of the title, and obtain the second initial probability of the title referencing a table based on the category of the title and a preset mapping relationship; the preset mapping relationship includes the mapping relationship between the category and the probability of referencing a table; S240, obtain the probability of the title reference table based on the first initial probability of the title reference table and the second initial probability of the title reference table.

3. The text data governance method based on a large model according to claim 2, characterized in that, S220 includes: S221, For any keyword that is successfully matched in the title and the content summary corresponding to the title, the product of the weight of the keyword and the number of times the keyword appears in the title and the content summary corresponding to the title is determined as the initial value of the keyword; S222, the initial values ​​of the keywords that are successfully matched in the title and the corresponding content summary are summed to obtain the intermediate value of the title; S223, adjust the median value of the title based on the length of the title and the content summary corresponding to the title to obtain the adjusted title result; S224, normalize the correction result of the title to obtain the first initial probability of the title referencing the table.

4. The text data governance method based on a large model according to claim 2, characterized in that, S240 includes: S241, Obtain the first corrected probability of the title reference table based on the first initial probability of the title reference table and the preset first weight; the preset first weight is greater than 0; S242, obtain the second corrected probability of the title reference table based on the second initial probability of the title reference table and the preset second weight; the preset second weight is greater than 0; S243, the sum of the first modified probability and the second modified probability is determined as the probability of the title referencing the table.

5. The text data governance method based on a large model according to claim 2, characterized in that, The category of the title is obtained using a predefined chapter type classifier, which is used to classify the title into a predefined category. In the preset mapping relationship, at least two different predefined categories have different probabilities of corresponding reference tables.

6. The text data governance method based on a large model according to claim 5, characterized in that, The predefined categories include Introduction, Background, Methods, Results, Discussion, Conclusion, and Other.

7. The text data governance method based on a large model according to claim 1, characterized in that, Matching each sentence corresponding to a candidate title in the main text with the summary description and data in the candidate table includes: S410, determine whether the specified sentence matches the summary description of the candidate table. If they match, proceed to S420; otherwise, determine that the specified sentence does not match the summary and data of the candidate table. The specified sentence is any sentence corresponding to the candidate title in the main text. S420, use the large model to reason and judge the data of the specified sentence and the candidate table, prompt the large model to judge whether the specified sentence is inferred based on the data of the candidate table, and output a binary judgment result of yes or no; S430, if the output is yes, then it is determined that the specified sentence matches the summary description and key data of the candidate table; otherwise, it is determined that the specified sentence does not match the summary and data of the candidate table.

8. The text data governance method based on a large model according to claim 1, characterized in that, The method further includes: if the content of a page in the target report text contains a table identifier, then displaying preset information corresponding to the table identifier in the footer of that page.

9. The text data governance method based on a large model according to claim 8, characterized in that, The method further includes: if the amount of information to be displayed at the footer position of a page in the target report text is greater than a preset information threshold, then the preset information corresponding to each identifier to be displayed at that footer position is simplified.

Images