Text information structured recovery method and system based on large language model and application

Abstract

The invention provides a text information structured recovery method and system based on a large language model and application. The text information structured recovery method comprises the following steps: S1, extracting original text content from a webpage or an unstructured document; s2, designing a cue word template according to different scenes and target structures, and generating cue words; s3, guiding a large language model to analyze the original text content and the cue word in the step S2, and generating a text result with a hierarchical structure; s4, analyzing a text result in the step S3, constructing a semantic structure tree, and forming a multi-layer nested structure; and S5, applying the multi-layer nested structure in the step S4 to database modeling and content indexing, compared with the defects of non-uniform structure loss, poor universality and semantic understanding intelligence deficiency in the prior art, the manual processing cost can be remarkably reduced, the data structuring efficiency and accuracy are improved, and the semantic understanding intellectuality is improved. And a stable and high-quality structured text support is provided for a large model ecological system.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a method, system and application for structured restoration of text information based on a large language model. Background Art

[0002] In recent years, with the breakthrough progress of large language models (LLMs) such as GPT, DeepSeek, and Claude in the field of natural language processing (NLP), model-based training and applications have been rapidly popularized in all walks of life. However, to ensure that large models have good semantic understanding and generation capabilities, high-quality, structured training datasets have become one of the fundamental and key elements. Especially in scenarios such as large model pre-training and fine-tuning, construction of RAG (retrieval-augmented generation) systems, and building of vertical knowledge bases, the importance of structured text data has become increasingly prominent.

[0003] Although existing web scraping and text collection technologies are relatively mature, there are still the following significant deficiencies when facing the task of restoring multi-source and messy text structures:

[0004] 1. Structure loss and non-uniformity: Most texts will lose the original structure information after collection or parsing, such as titles in HTML, paragraph relationships in document formats, or layout levels in OCR, resulting in a lack of clear logical organization in semantics; 2. Severe dependence on manual annotation: Traditional processing methods often rely on manual annotation and division of text structures, which not only consumes human costs but also easily introduces subjective biases and is difficult to standardize; 3. Poor generality of rule methods: Currently adopted rule matching (such as regular expressions), DOM node parsing, or templating methods are applicable to data with clear structure rules, but have poor effects on non-standard formats such as scanned documents and return values of dynamic content interfaces, and have weak generalization capabilities; 4. Lack of intelligent semantic understanding: Most existing methods rely on shallow syntactic features and cannot truly understand the semantic logic and hierarchical structure between texts, making it difficult to automatically construct a structured text representation with semantic accuracy. These technical bottlenecks severely restrict the batch acquisition of high-quality corpora, and thus affect the training effect of large models and the response quality of RAG systems.

[0005] Therefore, currently, the structured restoration methods and storage solutions with intelligent semantic understanding capabilities and the ability to automatically restore hierarchical structures still need to be further improved. Summary of the Invention

[0006] In view of this, the present invention proposes a method, system and application for structured restoration of text information based on a large language model, which solves the technical problems of structure loss and non-uniformity, poor generality, and lack of intelligent semantic understanding in the prior art.

[0007] The technical solution of the present invention is realized as follows:

[0008] In a first aspect, the present invention provides a method for structured recovery of text information based on a large language model, including the following steps:

[0009] S1, Extract the original text content from a web page or an unstructured document;

[0010] S2, Design a prompt template according to different scenarios and target structures to generate a prompt;

[0011] S3, Guide the large language model to parse the original text content and the prompt described in step S2 to generate a text result with a hierarchical structure;

[0012] S4, Parse the text result of step S3, construct a semantic structure tree, and form a multi-layer nested structure;

[0013] S5, Use the multi-layer nested structure described in step S4 for database modeling and content indexing.

[0014] On the basis of this technical solution, further preferably, the method further includes:

[0015] S6, Set up structure verification, mark and prompt abnormal structures, and fine-tune the results of some structures to improve the accuracy.

[0016] On the basis of this technical solution, further preferably, the text result described in step S3 includes a title - sub-title - paragraph content.

[0017] On the basis of this technical solution, further preferably, step S4 specifically includes the following steps:

[0018] Parse the text result of step S3, extract the hierarchical information of the title and the paragraph content, construct a semantic structure tree representing the hierarchical relationship, and form a multi-layer nested structure for index modeling and storage.

[0019] On the basis of this technical solution, further preferably, the database modeling described in step S5 includes the following steps:

[0020] Establish a title hierarchy table and a content table;

[0021] Establish a mapping relationship between the paragraph content of the content table and the title hierarchy;

[0022] Establish a retrieval index to support title query, hierarchical backtracking, and full-text keyword search.

[0023] In a second aspect, the present invention provides a system for structured recovery of text information based on a large language model, applying the method for structured recovery of text information based on a large language model according to any item in the first aspect, including:

[0024] A text collection module for extracting the original text content from web pages or unstructured documents;

[0025] A prompt engineering design module for designing prompt templates and generating prompts according to different scenarios and target structures;

[0026] A large language model parsing module that guides the large language model to parse the original text content and the prompts described in the prompt engineering design module, and generates a text result with a hierarchical structure;

[0027] A semantic structure tree module that parses the text result of the large language model parsing module, constructs a semantic structure tree, and forms a multi-level nested structure;

[0028] A database and index construction module that uses the multi-level nested structure described in the semantic structure tree module for database modeling and content indexing.

[0029] On the basis of this technical solution, further preferably, it further includes a structure verification module for verifying whether the system output meets the hierarchical format requirements, marking and prompting abnormal structures, and fine-tuning part of the structure results to ensure high accuracy.

[0030] In a third aspect, the present invention also provides an application of the method for structured recovery of text information based on a large language model described in the first aspect in a RAG retrieval system.

[0031] On the basis of this technical solution, further preferably, the present invention also provides an application of the method for structured recovery of text information based on a large language model described in the first aspect in knowledge graph construction.

[0032] On the basis of this technical solution, further preferably, the present invention also provides an application of the method for structured recovery of text information based on a large language model described in the first aspect in structured corpus management.

[0033] The method, system and application for structured recovery of text information based on a large language model described in the present invention have the following beneficial effects compared with the prior art:

[0034] Compared with the prior art, the semantic hierarchical structure recovery ability of the large language model of the present invention does not rely on manual processing, web page DOM structure or preset rules, but through the semantic understanding ability of the large language model, it can intelligently identify the logical structure relationship between the title and the text from the chaotic original text, and can achieve automatic hierarchical division (such as first-level title, second-level title, content, etc.);

[0035] Meanwhile, through the structured control output driven by prompt engineering, a dedicated prompt template (Prompt Engineering) is designed and used to guide the large language model to output multi-level headings and content structures in a specified format, making the structure restoration result consistent and controllable, and improving the system stability and expansion ability.

[0036] Moreover, the seamless mapping between the structured text and the database index structure directly maps the semantic structure tree output by the model to the hierarchical index structure of the database. By constructing the index relationship between the headings and the content, the efficient retrieval and content traceability capabilities of structured data are realized, which are applicable to RAG systems, knowledge base construction, and content management.

[0037] Finally, using the structure parsing and tree-like modeling mechanism, by parsing the format symbols output by the model, a semantic hierarchical tree is constructed to realize the "parent-child" node organization relationship between the headings, and then it is used for database structure generation and navigational indexing. It also supports highly general processing of unstructured input, does not rely on existing structure information, and directly acts on plain text or OCR recognition results to achieve unified structured collation of data with different sources and uneven quality. Brief Description of the Drawings

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a flowchart of the method for structured restoration of text information based on a large language model described in Embodiment 1 of the present invention;

[0040] Figure 2 It is a framework diagram of the system for structured restoration of text information based on a large language model described in Embodiment 2 of the present invention;

[0041] Figure 3 It is a schematic diagram of the semantic structure tree module described in Embodiment 2 of the present invention;

[0042] Figure 4 It is a framework diagram of the database index module described in Embodiment 2 of the present invention. Detailed Description of the Embodiments

[0043] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0044] Existing web scraping and text collection technologies still have significant deficiencies when facing the task of restoring multi-source and messy text structures. Therefore, how to extract and restore a clearly structured semantic hierarchy from a large amount of text with complex and disorderly sources is a core supporting technology in the construction of the large model ecosystem.

[0045] The present invention provides a method, system and application for structured restoration of text information based on a large language model. By designing appropriate prompt engineering, the large language model is guided to perform intelligent semantic parsing on text data with messy content, automatically identify the logical relationships between titles, sub-titles and content in the text, and generate structured text with a clear semantic hierarchy; construct a semantic structure tree with a title-content index structure, and then use the structured result for database modeling and content indexing to achieve efficient organization and management of text data.

[0046] This method has strong versatility and robustness, can be applied to various text contents with different sources and formats, and gets rid of the dependence on the original structure. At the same time, through the model's ability to understand and organize semantics, the original structure and logical relationships of the text can be effectively restored, greatly improving the data quality.

[0047] Through the method provided by the present invention, the manual processing cost can be significantly reduced, and the efficiency and accuracy of data structuring can be improved. It is especially suitable for scenarios such as large model training corpus preparation, fine-tuning data construction, and RAG knowledge base establishment, providing stable and high-quality structured text support for the large model ecosystem.

[0048] Example 1

[0049] A method for structured restoration of text information based on a large language model, as Figure 1 shown, includes the following steps:

[0050] S1, extract the original text content from a web page or an unstructured document;

[0051] S2, design a prompt template according to different scenarios and target structures, and generate a prompt;

[0052] S3, guide the large language model to parse the original text content and the prompt described in step S2, and generate a text result with a hierarchical structure;

[0053] S4. Analyze the text result of step S3, construct a semantic structure tree, and form a multi-level nested structure;

[0054] S5. Use the multi-level nested structure described in step S4 for database modeling and content indexing.

[0055] More specifically, the method further includes:

[0056] S6. Set up structure verification, mark and prompt abnormal structures, and fine-tune the results of some structures to improve accuracy.

[0057] Specifically, the hierarchical text result described in step S3 includes title - sub-title - paragraph content.

[0058] Specifically, step S4 specifically includes the following steps:

[0059] Analyze the text result of step S3, extract the hierarchical information of the title and the paragraph content, construct a semantic structure tree representing the hierarchical relationship, and form a multi-level nested structure for index modeling and storage.

[0060] Specifically, the database modeling described in step S5 includes the following steps:

[0061] Establish a title hierarchy table and a content table;

[0062] Establish a mapping relationship between the paragraph content of the content table and the title hierarchy;

[0063] Establish a retrieval index to support title query, hierarchical backtracking, and full-text keyword search.

[0064] In a preferred embodiment, the structured output of the large language model is as follows:

[0065] First, input the large language model, and the black bold text is the prompt word.

[0066] Table 1 Comparison of Structured Output of Large Language Model

[0067]

[0068]

[0069]

[0070]

[0071] Thus, it can be seen that the semantic structure can be accurately restored from messy text: the large model has the ability of context reasoning, can go beyond traditional rule-based methods, and accurately identify the hierarchical relationship between the title and the content.

[0072] Example 2

[0073] A text information structured recovery system based on a large language model, as Figure 2 shown, includes:

[0074] A text collection module, used to extract the original text content from web pages or unstructured documents;

[0075] Specifically, use web crawlers, API scraping, file parsing, or OCR character recognition, etc. to extract the original text content from web pages or other unstructured documents. Such texts usually show semantic mixing, chaotic logical order, and lack of clear hierarchical division.

[0076] A prompt engineering design module, used to design prompt templates and generate prompts according to different scenarios and target structures;

[0077] Specifically, design dedicated prompt templates (Prompt Engineering) for different data scenarios and target structures. The prompts guide the large language model to conduct structured understanding and hierarchical reorganization of the input content. For example, the model is prompted to identify content formats such as titles, subtitles, and body text in paragraphs and output them according to the preset hierarchical format. Design prompts in combination with the structured target format to guide the language model to output a standardized hierarchical structure.

[0078] In a preferred embodiment, the prompt examples are as follows:

[0079]

[0080] The prompts can be customized according to different fields (such as documents, encyclopedia articles, teaching materials, biology, medicine, computer, etc.) to enhance the structural output stability and semantic consistency of the model.

[0081] A large language model parsing module, guiding the large language model to parse the original text content and the prompts described in the prompt engineering design module, and generating a text result with a hierarchical structure;

[0082] Specifically, splice the above prompts and the original text and input them into the large language model (LLM). Based on its language understanding and context organization capabilities, the model outputs a text result with a hierarchical structure. The output format usually includes:

[0083] Multi-level headings (distinguishing levels such as first-level, second-level, third-level, etc.);

[0084] The body text content corresponding to each level of heading;

[0085] Natural paragraphs organized by semantics.

[0086] The semantic structure tree module parses the text results of the large language model parsing module and constructs a semantic structure tree, such as Figure 3 As shown, a multi-layer nested structure is formed;

[0087] Specifically, the text results returned by the model are parsed, title hierarchy information and paragraph content are extracted, and a semantic structure tree representing the hierarchical relationships is constructed. Each title node is connected to its corresponding child nodes and content nodes, forming a complete multi-layer nested structure that facilitates subsequent index modeling and storage.

[0088] The database and index building module uses the multi-layer nested structure described in the semantic structure tree module for database modeling and content indexing.

[0089] Specifically, based on the semantic structure tree, the database structure is designed and the storage operation is completed. Figure 4 As shown, database modeling includes the following steps:

[0090] 1. Create a title hierarchy table (first-level title, second-level title, etc.);

[0091] 2. Establish a mapping relationship between each paragraph content in the table of contents and its corresponding heading level;

[0092] 3. Establish a search index to support functions such as title query, hierarchical backtracking, and full-text keyword search.

[0093] Specifically, the system also includes a structure verification module, which is used to verify whether the system output meets the hierarchical format requirements, mark abnormal structures, and fine-tune some structural results to ensure high accuracy.

[0094] In a preferred embodiment, the method described in Example 1 is applied to efficient calling scenarios such as RAG retrieval system, high-dimensional knowledge graph construction, and structured corpus management, and can serve downstream AI system training and knowledge management: the sorted text can be directly used as high-quality structured corpus for large model training, fine-tuning, RAG knowledge retrieval, etc., significantly improving the model training efficiency and the response accuracy of the question-answering system.

[0095] The system automatically restores the text hierarchy through the model, which can achieve the following technical effects:

[0096] 1. Reduce labor costs and error rates: Complex text can be structured without human intervention, significantly saving manual annotation time and avoiding subjective misjudgment;

[0097] 2. High-precision restoration of semantic structure from cluttered text: The large model has contextual reasoning capabilities, surpassing traditional rule-based methods and accurately identifying the hierarchical relationship between titles and content.

[0098] 3. Significantly improved database indexing capabilities: Through structured title-content mapping, dual support for hierarchical navigation and keyword indexing is achieved, greatly enhancing the database retrieval efficiency;

[0099] 4. Adapt to different application scenarios and have versatility: Multi-format and multi-source texts, such as forum posts, news, Wikipedia, OCR results of scanned documents, etc., from web texts to documents, from news and information to book content, can all apply the method of the present invention for structure restoration and index reconstruction;

[0100] 5. Avoid dependence on web page structure and improve anti-interference ability: It does not depend on the HTML node structure and can restore the structure only relying on text semantics, which is suitable for complex web page environments with irregular data structures.

[0101] The present invention provides a method, system and application for structured restoration of text information based on large language models, which can significantly reduce the manual processing cost, improve the data structuring efficiency and accuracy, and is especially suitable for scenarios such as large model training corpus preparation, fine-tuning data construction, and RAG knowledge base establishment, providing stable and high-quality structured text support for the large model ecosystem.

[0102] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for structured recovery of text information based on a large language model, characterized in that It includes the following steps: S1. Extract the original text content from a web page or an unstructured document; S2. Design a prompt template according to different scenarios and target structures to generate a prompt; S3. Guide a large language model to parse the original text content and the prompt described in step S2 to generate a text result with a hierarchical structure; S4. Parse the text result of step S3, construct a semantic structure tree, and form a multi-level nested structure; S5. Use the multi-level nested structure described in step S4 for database modeling and content indexing.

2. The text information structured recovery method based on a large language model according to claim 1, wherein The method further includes: S6. Set up structure verification, mark and prompt abnormal structures, and fine-tune the results of some structures to improve the accuracy.

3. The method for structured recovery of text information based on a large language model according to claim 1, wherein The text result described in step S3 includes a title - sub-title - paragraph content.

4. The method for structured recovery of text information based on a large language model according to claim 3, wherein, Step S4 specifically includes the following steps: Parse the text result of step S3, extract the hierarchical information of the title and the paragraph content, construct a semantic structure tree representing the hierarchical relationship, and form a multi-level nested structure for index modeling and storage.

5. The method for structured recovery of text information based on a large language model according to claim 4, wherein The database modeling described in step S5 includes the following steps: Establish a title hierarchy table and a content table; Establish a mapping relationship between the paragraph content of the content table and the title hierarchy; Establish a retrieval index to support title query, hierarchical backtracking, and full-text keyword search.

6. A text information structured recovery system based on a large language model, which applies the text information structured recovery method based on a large language model described in any one of claims 1-5, and is characterized in that, It includes: A text collection module for extracting the original text content from a web page or an unstructured document; A prompt engineering design module for designing a prompt template according to different scenarios and target structures to generate a prompt; A large language model parsing module for guiding a large language model to parse the original text content and the prompt described in the prompt engineering design module to generate a text result with a hierarchical structure; A semantic structure tree module for parsing the text result of the large language model parsing module, constructing a semantic structure tree, and forming a multi-level nested structure; A database and index construction module for using the multi-level nested structure described in the semantic structure tree module for database modeling and content indexing.

7. The text information structured recovery system based on a large language model according to claim 6, wherein It further includes a structure verification module for verifying whether the system output meets the hierarchical format requirements, marking and prompting abnormal structures, and fine-tuning some structure results to ensure high accuracy.

8. Application of the method for structured recovery of text information based on a large language model according to any one of claims 1-5 in a RAG retrieval system, knowledge graph construction, and structured corpus management.

Images