Text structured information generation method based on large model

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a text structured information generation method based on a large model. The method comprises the steps that a data set is prepared, a first large model is finely adjusted, and a structured mode is generated; the data set preparation mode comprises a man-machine combination mode and a large model reverse text generation mode; introducing an RAG mechanism to correct the structured mode generated by the first large model to obtain a complete structured mode; mode filling is carried out on the complete structured mode, the structured text is comprehensively applied through fine adjustment, retrieval, enhancement and generation and other technologies, and the method has remarkable advantages in the aspects of dynamic structure generation capacity, result accuracy and processing efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a method for generating text structured information based on a large model. Background Art

[0002] With the rapid development of the internet and big data technologies, massive amounts of unstructured text data, such as news reports, social media posts, research papers, and product reviews, are constantly emerging. This text data contains rich information, but due to its unstructured nature, it is difficult to directly use in applications such as data analysis, knowledge graph construction, and information retrieval. Therefore, extracting valuable information from this text data and converting it into structured data has become a key research topic in the field of natural language processing (NLP).

[0003] Traditional text structuring methods primarily rely on predefined patterns or rules. For example, rule-based systems use hand-written rules to identify and extract specific types of information, such as entity recognition and relationship extraction. However, this approach has the following limitations: a. High rule-writing cost: Writing and maintaining rules requires significant manpower and time, and it is difficult to cover all possible text patterns. b. Difficulty processing complex text: Natural language text is highly diverse and complex, and predefined rules often struggle to handle complex sentence structures and semantic relationships. c. Poor generalization: Rule-based methods typically only perform well in specific domains or datasets and are difficult to generalize to other domains or datasets. In recent years, deep learning technology has made significant progress in the field of natural language processing. In particular, large models based on the Transformer architecture (such as BERT and GPT) have performed well in tasks such as text understanding and generation. These large models are pre-trained on large-scale corpora to capture rich language patterns and semantic information. However, despite the excellent performance of large models in text understanding, there are still some challenges when generating structured information: a. Hallucination problem: When generating structured information, large models may generate information that does not match reality, which is the so-called "hallucination" problem. For example, the model may incorrectly identify entity types, relationship types, or event types, resulting in inaccurate generated structured information. b. Lack of structured constraints: When generating structured information, large models lack clear constraints on structured patterns, resulting in the generated structured information may not conform to the expected format or semantics. c. Strong data dependence: The performance of large models is highly dependent on the quality and diversity of training data. If certain types of information or patterns are lacking in the training data, the model may perform poorly in generating the corresponding structured information. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a method for generating text structured information based on a large model, which includes:

[0005] S1: Prepare the dataset and fine-tune the first large model to generate structured patterns; methods for preparing the dataset include human-computer interaction and reverse text generation from the large model;

[0006] S2: Introducing the RAG mechanism to correct the structured pattern generated by the first model to obtain a complete structured pattern;

[0007] S3: Fill in the complete structured pattern to obtain structured text.

[0008] Preferably, the process of preparing the data set using a human-computer combination approach includes:

[0009] Step 1: Prepare a small initial dataset, including natural language text and its corresponding structured patterns;

[0010] Step 2: Use the dataset to fine-tune the first model to obtain an intermediate model;

[0011] Step 3: Obtain new natural language text and use the intermediate model to perform pattern generation tasks on the new natural language text to generate preliminary structured patterns;

[0012] Step 4: Manually correct the structured pattern generated by the intermediate model and modify the incorrectly labeled parts;

[0013] Step 5: Add the corrected structured pattern and the corresponding natural language text to the dataset to accumulate more data;

[0014] Step 6: Repeat steps 2 to 5, gradually accumulating data until a fully fine-tuned first model is achieved.

[0015] Preferably, the process of preparing a data set by using a large model to reversely generate text includes:

[0016] Step 1: Prepare a batch of structured schema data, including entity types, relationship types, and event types;

[0017] Step 2: Use the first model to reversely generate natural language text based on the structured pattern data;

[0018] Step 3: Manually correct the generated natural language text to ensure its accuracy and naturalness;

[0019] Step 4: Add the corrected natural language text and the corresponding structured pattern to the dataset to accumulate more data;

[0020] Step 5: Repeat steps 2 to 4 to gradually accumulate data until the preset data volume is reached.

[0021] Preferably, the process of introducing the RAG mechanism to correct the structured pattern generated by the first large model includes: constructing a RAG document to obtain predefined entity types, relationship types and event types; the RAG mechanism constrains and corrects the structured pattern generated by the large model according to the RAG document to ensure that the generated entity types, relationship types and event types comply with the predefined specifications.

[0022] Furthermore, the process of building a RAG document includes:

[0023] Knowledge collection: Collect domain knowledge, expert knowledge, or entity types, relationship types, and event type information from existing datasets;

[0024] Knowledge organization: organize the collected knowledge to form a complete RAG document;

[0025] Knowledge verification: Verify the organized RAG documents to ensure the accuracy and completeness of the knowledge.

[0026] Preferably, the process of pattern filling the complete structured pattern includes:

[0027] Obtain the text to be structured and splice it with the complete structured pattern to obtain splicing information;

[0028] The second largest model is used to reason about the spliced ​​information and generate structured text.

[0029] Furthermore, the structured text is a completely structured JSON format text.

[0030] The present invention provides the following benefits: It designs a method for generating structured text information based on a large model. This approach separates pattern generation from pattern filling, facilitating the optimization and improvement of each. Furthermore, it allows for flexible adjustments to pattern generation rules and filling strategies to accommodate complex text information extraction requirements in diverse scenarios. Through the integrated application of fine-tuning and retrieval-enhanced generation techniques, the present invention demonstrates significant advantages in dynamic structure generation capabilities, result accuracy, and processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flowchart of the method for generating text structured information based on a large model in the present invention;

[0032] Figure 2 The adaptive pattern structure diagram generated for the large model in the present invention;

[0033] Figure 3Schematic diagram of data enhancement by reversely generating text from a large model in the present invention;

[0034] Figure 4 This is a schematic diagram of data enhancement using a human-machine combination approach in the present invention;

[0035] Figure 5 Schematic diagram of the input structured form of pattern filling in the present invention;

[0036] Figure 6 This is a schematic diagram of the final structured form output after pattern filling in the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] The present invention proposes a method for generating text structured information based on a large model, such as Figure 1 As shown, the method includes the following contents:

[0039] S1: Prepare the dataset and fine-tune the first model to generate structured patterns.

[0040] First, we construct the instruction fine-tuning dataset. The content of the instruction fine-tuning dataset is divided into two parts: input and output. The input is the natural language text to be structured, and the output is the corresponding structured schema. The structured schema is represented in JSON format and contains information such as entity type, relationship type, and event type. For example, the text "Steve Jobs became the CEO of Apple in 1997". Its structured schema is as follows Figure 2 shown.

[0041] The present invention can adopt two data enhancement strategies to prepare the data set, namely the human-computer combination method and the large model reverse text generation method.

[0042] The large model reverse text generation method uses the large model to reversely generate natural language text based on structured pattern data, and then manually corrects the generated text and adds it to the dataset. Figure 3 As shown, the specific steps are:

[0043] Step 1: Prepare a batch of structured schema data, including entity types, relationship types, and event types;

[0044] Step 2: Use the first language model to reversely generate natural language text based on the structured pattern data;

[0045] Step 3: Manually correct the generated natural language text to ensure its accuracy and naturalness;

[0046] Step 4: Add the corrected natural language text and the corresponding structured pattern to the dataset to accumulate more data;

[0047] Step 5: Repeat steps 2 to 4 to gradually accumulate data until the preset data volume is reached.

[0048] After preparing a dataset using a large model for reverse text generation, fine-tuning the model is required. Model fine-tuning is a technique that uses a task-specific dataset to further train an existing pre-trained model. This approach allows the model to adapt to the nuances of a specific task while maintaining the general knowledge it has learned from a large dataset. Using a fine-tuned model, you can achieve the following benefits:

[0049] a. Improve performance: Fine-tuning can significantly improve the performance of the model on specific tasks.

[0050] b. Reduced training time: Compared to training a model from scratch, fine-tuning usually requires less training time and computing resources.

[0051] c. Adaptation to specific domains: Fine-tuning can help the model better adapt to data and tasks in specific domains. Qwen2.5-72B-Instruct was selected as the base model. It is one of the latest large language model series released by Alibaba Cloud. The 72B model has significantly improved capabilities in areas such as coding and mathematics. It supports inputs of up to 128K tokens and can generate long texts of more than 8K tokens. The model also provides multilingual support, covering more than 29 languages, including Chinese, English, etc. The model has significantly improved in following instructions, understanding structured data, and generating structured output (especially JSON). The fine-tuning process is as follows:

[0052] a. Data preprocessing: Convert annotated datasets into model input formats, such as converting text and structured patterns into token sequences. Clean and standardize the data to ensure data quality.

[0053] b. Model loading: Load the pre-trained large model, use the LoRA (Low-Rank Adaptation) fine-tuning strategy, and initialize the LoRA parameters.

[0054] c. Model training: Use the labeled dataset to train the model and optimize the model parameters. Adopt the LoRA fine-tuning strategy, only updating the LoRA parameters while keeping the pre-trained model parameters unchanged to reduce computing resources and memory consumption.

[0055] d. Model Validation: Use the validation set to validate the model, evaluate its performance, and make necessary adjustments. Monitor the model's performance on the validation set to ensure its accuracy and stability in generating structured patterns.

[0056] The method of building a dataset in a human-machine combination is to first prepare a part of the dataset to fine-tune an intermediate model, then directly let it perform the pattern generation task, and then manually modify the incorrectly labeled parts. After accumulating more data, fine-tune the model again, and then repeat the previous steps to finally achieve a fully fine-tuned model, such as Figure 4 As shown, the implementation steps are:

[0057] Step 1: Prepare a small initial dataset, including natural language text and its corresponding structured patterns;

[0058] Step 2: Use the dataset to fine-tune the first model to obtain an intermediate model;

[0059] Step 3: Obtain new natural language text and use the intermediate model to perform pattern generation tasks on the new natural language text to generate preliminary structured patterns;

[0060] Step 4: Manually correct the structured pattern generated by the intermediate model and modify the incorrectly labeled parts;

[0061] Step 5: Add the corrected structured pattern and the corresponding natural language text to the dataset to accumulate more data;

[0062] Step 6: Repeat steps 2 to 5, gradually accumulating data until a fully fine-tuned first model is achieved.

[0063] In the process of building the data set in a human-machine combined way, the large model is also fine-tuned. As the data set is continuously accumulated, the large model is also fine-tuned. When the data set is prepared, the large model is also fine-tuned.

[0064] S2: The RAG mechanism is introduced to correct the structured pattern generated by the first model to obtain a complete structured pattern.

[0065] In the model reasoning phase, the Retrieval Augmented Generation (RAG) mechanism is introduced. The RAG mechanism retrieves information such as predefined entity types, relationship types, and event types by retrieving external knowledge bases or documents.

[0066] The role of the RAG mechanism is to constrain and correct the structured patterns generated by large models, ensuring that the generated entity types, relationship types, and event types conform to predefined specifications, thereby reducing hallucination problems.

[0067] RAG document construction: Build a RAG document containing predefined entity types, relationship types, and event types. RAG documents can be built based on domain knowledge, expert knowledge, or existing datasets.

[0068] RAG document construction process:

[0069] 1) Knowledge collection: Collect domain knowledge, expert knowledge, or information such as entity types, relationship types, and event types from existing datasets.

[0070] 2) Knowledge organization: Organize the collected knowledge to form a complete RAG document.

[0071] 3) Knowledge verification: Verify the organized RAG documents to ensure the accuracy and completeness of the knowledge.

[0072] Simple example of a RAG document:

[0073] {

[0074] "Entity Type": ["Organization", "Task", "Location", "Country"],

[0075] "Relationship Type": ["Worked at", "Born at", "Died at"],

[0076] "Event Type": ["Appointment", "Issue", "Natural Disaster"]

[0077] }

[0078] RAG mechanism application:

[0079] During model inference, the large model is first used to generate a preliminary structured schema. The RAG mechanism is then used to retrieve RAG documents to obtain predefined information such as entity types, relationship types, and event types. Finally, the structured schema generated by the large model is corrected based on the information in the RAG documents to ensure that it conforms to predefined specifications.

[0080] S3: Fill in the complete structured pattern to obtain structured text.

[0081] The main goal of pattern filling is to extract specific entity, relationship, and event information from the original text based on the structured schema output by the pattern generation module, and fill it into the structured schema, ultimately outputting a fully structured JSON-formatted text. This invention adopts a method of directly using a large model for pattern filling. The input of the large model is the text to be structured and its schema, and the output is structured information. The process of pattern filling is as follows:

[0082] Large model input: Get the text to be structured and splice it with the complete structured pattern to obtain splicing information; Figure 5 shown.

[0083] Large model reasoning: Use the second largest model to reason about the spliced ​​information and generate structured text.

[0084] Based on the input original text and structured pattern, the large model extracts specific entity, relationship and event information from the text and fills it into the structured pattern.

[0085] The structured information output by the large model is a fully structured Json format text, such as Figure 6 shown.

[0086] In summary, this paper presents a method for generating structured text information based on a large model. This approach separates pattern generation from pattern filling, facilitating the optimization and improvement of each. Furthermore, by flexibly adjusting pattern generation rules and filling strategies, it can adapt to the complex text information extraction needs of various scenarios. Through the integrated application of fine-tuning and retrieval-enhanced generation techniques, this paper demonstrates significant advantages in dynamic structure generation capabilities, result accuracy, and processing efficiency.

[0087] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating text structured information based on a large model, characterized in that: include: S1: Prepare the dataset and fine-tune the first model to generate structured patterns; Methods for preparing datasets include human-computer integration and large-scale model reverse text generation; S2 : The RAG mechanism is introduced to correct the structured pattern generated by the first model and obtain a complete structured pattern; S3: Fill in the complete structured pattern to obtain structured text.

2. A method for generating text structured information based on a large model according to claim 1, characterized in that: The process of preparing a dataset using a human-computer combined approach includes: Step 1: Prepare a small initial dataset, including natural language text and its corresponding structured patterns; Step 2: Use the dataset to fine-tune the first model to obtain an intermediate model; Step 3: Obtain new natural language text and use the intermediate model to perform pattern generation tasks on the new natural language text to generate preliminary structured patterns; Step 4: Manually correct the structured pattern generated by the intermediate model and modify the incorrectly labeled parts; Step 5: Add the corrected structured pattern and the corresponding natural language text to the dataset to accumulate more data; Step 6: Repeat steps 2 to 5, gradually accumulating data until a fully fine-tuned first model is achieved.

3. The method for generating text structured information based on a large model according to claim 1, characterized in that: The process of preparing a dataset using a large model to reverse generate text includes: Step 1: Prepare a batch of structured schema data, including entity types, relationship types, and event types; Step 2: Use the first model to reversely generate natural language text based on the structured pattern data; Step 3: Manually correct the generated natural language text to ensure its accuracy and naturalness; Step 4: Add the corrected natural language text and the corresponding structured pattern to the dataset to accumulate more data; Step 5: Repeat steps 2 to 4 to gradually accumulate data until the preset data volume is reached.

4. The method for generating text structured information based on a large model according to claim 1, characterized in that: The process of introducing the RAG mechanism to correct the structured pattern generated by the first large model includes: constructing a RAG document to obtain predefined entity types, relationship types and event types; the RAG mechanism constrains and corrects the structured pattern generated by the large model according to the RAG document to ensure that the generated entity types, relationship types and event types comply with the predefined specifications.

5. A method for generating text structured information based on a large model according to claim 4, characterized in that: The process of building a RAG document includes: Knowledge collection: Collect domain knowledge, expert knowledge, or entity types, relationship types, and event type information from existing datasets; Knowledge organization: organize the collected knowledge to form a complete RAG document; Knowledge verification: Verify the organized RAG documents to ensure the accuracy and completeness of the knowledge.

6. The method for generating text structured information based on a large model according to claim 1, characterized in that: The process of pattern filling for a complete structured pattern includes: Obtain the text to be structured and splice it with the complete structured pattern to obtain splicing information; The second largest model is used to reason about the spliced ​​information and generate structured text.

7. A method for generating text structured information based on a large model according to claim 6, characterized in that: Structured text is fully structured text in JSON format.

Images