Text structuring method and system based on large language model

Abstract

The invention belongs to the technical field of information extraction, and discloses a text structuring method and system based on a large language model. The method comprises the following steps: performing training data processing by analyzing a JSON format of a large language model; performing training and fine tuning on the large language model, performing training by using vLLM, and deploying the fine-tuned large language model; and performing text structuring on the deployed large language model by adopting a CoT prompt mode, and further improving the structured output capability of the large language model by adopting a reflection strategy of a JSON analysis result for wrong JSON format data. According to the method, the problem that JSON character strings generated by a large model in a knowledge extraction task do not meet requirements is solved through combination of fine adjustment and instructions; and the large model generates a JSON (JavaScript Object Notation) of a knowledge graph specification according to a schema. Org requirement, so that end-to-end text structuring is realized, further knowledge processing is not needed, and the workload of subsequent knowledge processing is saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of information extraction, and in particular relates to a text structuring method and system based on a large language model. Background Art

[0002] The results of traditional open domain information extraction technology are highly uncontrollable and cannot be extracted according to user requirements; closed domain information extraction relies heavily on the construction of labeled data sets, focusing on entity extraction, relationship extraction or integrated extraction of these tasks. The subsequent processing flow is complex, and end-to-end text structuring cannot be achieved, and the migration capability is limited; knowledge extraction based on large models greatly reduces the dependence on labeled data, but due to the influence of fine-tuning, the large model will focus on the generation of actual text content and ignore the processing of punctuation marks when formatting the output based on the context, resulting in JSON formatting failure; another solution is to embed formatting syntax checks in the large model, such as the gramar tool, to achieve the purpose of improving formatting generation. This method will reduce the effect of knowledge extraction and the efficiency of large model reasoning, which is particularly obvious when the input text content is long.

[0003] Through the above analysis, the problems and defects of the existing technology are as follows: In the existing technology, the large model can generate a well-structured JSON format string, but in the actual knowledge extraction task, it is very difficult to extract knowledge according to the input text according to the instructions and output it in JSON format. Moreover, the large model does not meet the requirements for generating JSON strings in the knowledge extraction task, and further knowledge processing is required, which increases the workload of subsequent knowledge processing. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a text structuring method and system based on a large language model. The purpose of the present invention is to improve the ability of extracting structured knowledge from a large model by adjusting the loss function and performing fine-tuning for specific tasks, combined with prompt design.

[0005] The technical solution is as follows: A text structuring method based on a large language model, comprising:

[0006] S1, determining the hierarchical structure, key-value pair characteristics, and data types of the JSON format by analyzing the JSON format of the large language model;

[0007] S2, data training based on the hierarchical structure, key-value pair characteristics, and data types determined by the JSON format;

[0008] S3, fine-tuning the large language model to improve its ability to extract domains and output JSON formatting;

[0009] S4, fine-tuning of large language model deployment using vLLM;

[0010] S5, using the CoT prompt method to use the deployed large language model to structure the text. For JSON format data with errors, the reflection strategy of JSON parsing results is adopted to improve the structured output capability of the large language model.

[0011] In step S1, the string in JSON format is:

[0012] {"aaa":"bbb","ccc":["ddd","eee"],"fff":{"ggg":"hhh"}}

[0013] Among them, square brackets and curly brackets appear in pairs, strings are in double quotes, key-value pairs are separated by commas, and key values ​​are separated by colons;

[0014] The large language model generates the current character according to the above context, and each character is generated by selecting the character according to probability.

[0015] In step S2, data training is performed, including: crawling example rows from schema.org to form a seed set to form an information extraction dataset for a specified task; combining existing public information extraction datasets to form a multi-task learning dataset, and identifying and correcting the JSON format from different tasks.

[0016] In step S3, the large language model is fine-tuned, including: using a weighted cross entropy loss function to give different weights to the prediction results of certain specific JSON punctuation characters in the text structuring task to get more attention; the weighted cross entropy loss function is expressed as:

[0017]

[0018] In the formula, θ is the parameter of the model, x is the input sequence; y is the actual label, that is, the next character in the sequence; w y is the weight of the previous sample y, which is the element corresponding to the current sample selected from the weight vector w; C is the character set, [y=c] is the indicator function, which takes the value of 1 when y=c, otherwise it is 0; pθ(y|x) is the probability of predicting the target y when the model is given an input x; w is the weight vector, and the elements of the weight vector correspond to the weight of each category in C.

[0019] The fine-tuning includes: fine-tuning on the base model based on a weighted cross entropy loss function to improve the large language model's ability to extract domains and its ability to output JSON formatting.

[0020] In step S5, the deployed large language model is used to structure the text using CoT prompts, and few-shot is used to improve the structuring effect.

[0021] Step S5 specifically includes:

[0022] S5.1, Identity recognition: In command editing, when interacting with the large language model, a role or identity is set for the large language model through a preset identity, and the model responds and acts according to the preset identity during the conversation or task execution; the way to preset the identity is to specify the role played by the large model through text;

[0023] S5.2, Output format: In the JSON format output instructions of the large language model, the requirements of JSON format syntax and punctuation are emphasized;

[0024] S5.3, using CoT prompts to enable large language models to generate intermediate reasoning steps, improve the extraction performance of large language models in complex tasks, and increase the interpretability of large language model outputs;

[0025] S5.4, Reflection: Directly call the program to parse the JSON result output by the large language model. If the parsing fails, capture the cause of the exception and feed it back to the large language model, allowing the large language model to reflect and regenerate the parsing result based on the cause of the error, further improving the JSON output capability.

[0026] In step S5.3, the CoT prompt includes:

[0027] Problem definition: Identify the multi-step problem to be solved;

[0028] Problem decomposition: Breaking down a problem into smaller tasks or problems;

[0029] Create prompts: Create prompts for each small task to guide the large language model to reason;

[0030] Execution: Prompts are fed to a large language model in sequence and the output of each step is collected;

[0031] Analyze and optimize: Evaluate the output and optimize the prompts if necessary.

[0032] Another object of the present invention is to provide a text structuring system based on a large language model, the system implementing the text structuring method based on a large language model, the system comprising:

[0033] A JSON format analysis module, used to determine the hierarchical structure, key-value pair characteristics and data types of the JSON format by analyzing the JSON format of the large language model;

[0034] A training data processing module is used to perform data training based on the hierarchical structure, key-value pair characteristics, and data types determined in the JSON format;

[0035] Large language model training module, used to fine-tune the large language model, improve the large language model's ability to extract domains, and improve its ability to output in JSON format;

[0036] A deployment module for fine-tuning large language model deployments using vLLM;

[0037] The CoT prompt module is used to use the deployed large language model to structure text using the CoT prompt. For JSON format data with errors, the reflection strategy of JSON parsing results is adopted to improve the structured output capability of the large language model.

[0038] Furthermore, the system is mounted on a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can realize the functions of the above-mentioned text structuring system based on a large language model.

[0039] Combining all the above technical solutions, the beneficial effects of the present invention are as follows: the present invention solves the problem that the JSON string generated by the large model in the knowledge extraction task does not meet the requirements by combining fine-tuning with instructions; the large model generates JSON in accordance with the knowledge graph specification as required by schema.org, realizing end-to-end text structuring without the need for further knowledge processing, thus saving the workload of subsequent knowledge processing.

[0040] Using a weighted cross entropy loss function to fine-tune the large model can force the large model to pay more attention to the generation of characters related to the JSON format syntax when structuring the text, thereby improving the usability of the large model in generating JSON format strings; by presetting the identity, it is beneficial for the large model to perform targeted processing in the specified field and improve the processing effect; by using CoT to implement strict instruction requirements and extraction step guidance, the large model is prompted to perform reasoning and thinking, thereby improving the large model's content extraction effect; by emphasizing the output requirements for the JSON format in the prompt, the JSON formatted output capability of the large model is improved, and through few-shot, the extraction effect and formatted output capability of the large model are improved. Combined with the reflection strategy of the JSON parsing results, the formatted output capability can be further improved.

[0041] The traditional text structuring methods used in the industry, such as named entity recognition and relationship extraction, have problems such as task fragmentation, lengthy processes, and information fragmentation, which have always seriously restricted the application of text structuring tasks. Although the use of large models for text structuring has reduced the severity of the above problems, the output format of the large model cannot guarantee good structuring requirements, nor can it be directly applied to the knowledge graph. The technical solution of the present invention ensures compliance with the schema.org specification from the fine-tuning stage, can interact directly with the knowledge graph, and greatly reduces the post-processing work of importing knowledge into the knowledge graph; at the same time, the weighted cross entropy fine-tuning method is adopted, combined with the prompt word strategy, to maximize the possibility of improving the structured output capability of the large model and narrow the gap between text structuring and knowledge application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0043] Figure 1 is a flow chart of a text structuring method based on a large language model provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of a text structuring system based on a large language model provided by an embodiment of the present invention;

[0045] In the figure: 1. JSON format analysis module; 2. Training data processing module; 3. Large language model training module; 4. Deployment module; 5. CoT prompt module. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0047] Embodiment 1, as Figure 1 As shown, the text structuring method based on the large language model provided by the embodiment of the present invention includes:

[0048] S1, determining the hierarchical structure, key-value pair characteristics, and data types of the JSON format by analyzing the JSON format of the large language model;

[0049] S2, data training based on the hierarchical structure, key-value pair characteristics, and data types determined by the JSON format;

[0050] S3, fine-tuning the large language model to improve its ability to extract domains and output JSON formatting;

[0051] S4, fine-tuning of large language model deployment using vLLM;

[0052] S5, using the CoT prompt method to use the deployed large language model to structure the text. For JSON format data with errors, the reflection strategy of JSON parsing results is adopted to improve the structured output capability of the large language model.

[0053] Exemplarily, in step S1, JSON format analysis: the present invention performs an in-depth analysis on the JSON format output of the large model to clarify its hierarchical structure, key-value pair characteristics, and data types.

[0054] JSON string is a string with strict syntax, such as:

[0055] {"aaa":"bbb","ccc":["ddd","eee"],"fff":{"ggg":"hhh"}}

[0056] Among them, square brackets and curly brackets must appear in pairs, strings must be in double quotes, key-value pairs are separated by commas, and key values ​​are separated by colons.

[0057] Any punctuation error will cause the parsing of the JSON string to fail.

[0058] The essence of a language model is a probabilistic model that estimates the likelihood of the next word or other unit in a given sequence. Its core function is to predict the probability of the next word in a sequence after receiving a series of words. Specifically, a language model can calculate the probability of a sentence, which represents the likelihood of generating the sentence according to natural language rules.

[0059] Large Language Models (LLMs) are also language models. They are an artificial intelligence technology based on deep learning. They use multi-layer neural networks to understand and generate human language. These models are usually trained on massive amounts of text data to learn the statistical laws of language, including vocabulary selection, grammatical structure, and contextual relationships. First, input processing is performed: converting text into a form that the computer can understand, such as mapping words to vector space through word embedding.

[0060] Then the model is inferred: the input data is processed using a neural network to predict the probability distribution of the next word based on the previous words. Finally, the output is generated: the most likely word is selected as the output based on the predicted probability distribution, or multiple possible results are generated by random sampling. As can be seen, the large model has a certain degree of randomness when generating characters, especially, it may generate incorrect punctuation marks, such as Chinese double quotes or commas, or generate unmatched brackets. This leads to parsing failure.

[0061] In step S2, training data processing: Currently, there is no public data set that meets the specifications of schema.org. Therefore, the present invention forms a seed set by crawling example rows of schema.org, thereby forming an information extraction data set for a specified task; at the same time, combined with existing public information extraction data sets (such as named entity recognition, relationship extraction, event extraction, etc.), a multi-task learning data set is formed, which ensures the migration ability of large models for knowledge extraction and improves the ability to recognize and correct JSON formats from different tasks.

[0062] In step S3, the model is trained:

[0063] S301, loss function design, the cross entropy loss function allows different weights to be given to the prediction results of certain specific categories, thereby affecting the training process. In text structuring tasks, if you want to pay more attention to errors in certain specific JSON punctuation characters (such as sentence end markers), you can achieve this by adjusting the weights of these characters.

[0064] The weighted cross entropy loss function can be formally expressed as:

[0065]

[0066] In the formula, θ is the parameter of the model, x is the input sequence; y is the actual label, that is, the next character in the sequence; w y is the weight of the previous sample y, which is the element corresponding to the current sample selected from the weight vector w; C is the character set, [y=c] is the indicator function, which takes the value of 1 when y=c, otherwise it is 0; pθ(y|x) is the probability of predicting the target y when the model is given an input x; w is the weight vector, and the elements of the weight vector correspond to the weight of each category in C.

[0067] S302, Fine-tuning, Fine-tuning is the process of further training a pre-trained large model using a dataset in a specific field. It aims to optimize the performance of the model on a specific task so that the model can better adapt to and complete tasks in a specific field.

[0068] Based on the weighted cross entropy loss function in the previous section, the present invention adopts fine-tuning on the base model (there is no requirement for the specific selection of the base model, and it is also a large model after pre-training. In order to ensure the Chinese processing effect, the Chinese large model is preferred, such as Qwen2-7B, and the fine-tuning method can be full parameter fine-tuning or parameter efficient fine-tuning). Fine-tuning can improve the model's ability to extract the field, and more importantly, improve the output capability of JSON formatting. The full name of the fine-tuning (Fine-tuning) described in the present invention is supervised fine-tuning (Supervised Fine-tuning, SFT), which is essentially training. The principle is to use the pre-trained model as the starting point, and continue training on the data set of a specific task or field, so that the model can better adapt to this specific task or field. The core idea is transfer learning, that is, using the knowledge obtained on large-scale general data to accelerate and improve the learning of specific tasks.

[0069] In step S4, deployment: To improve the reasoning efficiency, the present invention uses vLLM to deploy the large language model. Specifically, we use an API access method compatible with OpenAI to deploy the large model application. After installing and configuring the vLLM environment, the large model can be deployed using the following command:

[0070] vllm serve Qwen / Qwen2-7B-Instruct--api-key xxx

[0071] Among them: "Qwen / Qwen2-7B-Instruct" is the model name, and api-key is the key required to access the API.

[0072] After deployment is complete, you can access the large model service by calling Python's OpenAI API.

[0073] In step S5, prompt design: adopt a multi-round dialogue mode, in which instructions such as identity recognition and output format requirements are provided, and the Chain of Thought (CoT) method is adopted to allow the large language model to structure the text according to the steps, and then combine few-shot to improve the structuring effect. We use the form of multi-round dialogue to embed few-shot into the dialogue as a logic, such as “[{“role”:”system”,”content”:”{system prompt}”},{“role”:”user”,“content”:”{example_input},{example_prompt}”},{“role”:”assistant”,”content”:”{example_output}”}…,{“role”:”user”,“content”:”{real_input}”}]” Finally, according to the JSON parsing error results, let the large language model reflect on the results to further improve the structured output capabilities of the large language model. Specifically include:

[0074] S5.1, identity recognition: In command editing, when interacting with the large language model through a preset identity, a role or identity is set for the large language model, and responses and actions are performed according to the preset identity during dialogue or task execution; the way to preset the identity is to indicate the role played by the large model through text, such as "You are an information extraction expert. Your task is to extract specified information from the input text according to the structured requirements (schema) given by the user." In the environment of the present invention, the preset identity is preferably "information extraction expert, knowledge graph structuring engineer", etc.

[0075] S5.2, Output format: The large language model is a probability-based generation model. In order to further enhance the ability of the large language model to output JSON format, the output requirements of the JSON format are further emphasized in the instructions. Specifically, the requirements for JOSN format syntax and punctuation are emphasized. That is, the following content is clearly required in the prompt: "Please output the final result in JSON format. "Key-value pairs" are separated by English colons ":", and multiple "key-value pairs" are separated by English commas ", ".

[0076] S5.3, CoT: Chain-of-Thought (CoT) is a technique to improve the performance of large language models (LLMs) on complex reasoning tasks. It significantly improves the reasoning ability of large language models by generating a series of intermediate reasoning steps. CoT prompts are different from regular prompts in that they encourage large language models to generate intermediate reasoning steps instead of directly providing answers. Doing so can improve the extraction performance of large language models on complex tasks such as arithmetic, common sense, and symbolic reasoning, and increase the interpretability of the output of large language models.

[0077] Exemplarily, the CoT prompt words are as follows:

[0078] "Your task is to determine whether the input text meets the extraction requirements based on the given pattern, and extract entities and attributes that match the user's requirements. The single-value attribute is a string, and the multi-value attribute is output as a list.

[0079] We will extract knowledge step by step according to the following process and give the thinking process:

[0080] 1) Carefully read the text entered by the user, understand and summarize the text content;

[0081] 2) Extract knowledge based on the specific description of the attribute. When extracting, first repeat the meaning of the attribute, and then extract the attribute in the input text based on the attribute meaning.

[0082] 3) Check if the extracted attribute contains the entity name. If not, return a null value. The null value answer format is "\nThe final answer is {}".

[0083] 4) If the entity name is extracted, the entity name and other attributes are output in JSON format. The answer format is "\nThe final answer is xxx"

[0084] First, clarify the task requirements and make judgments before extraction, which is step 1;

[0085] Then guide the model to extract step by step according to the process, which is step 2;

[0086] Then the model is required to understand the input text in order to better extract the content, which is step 3;

[0087] Then subdivide the task and require the model to extract the given attributes one by one, while explaining the meaning and requirements of the attributes. This is step 4.

[0088] Then guide the model to perform reasoning (that is, the big model determines whether to return a result), which is step 5;

[0089] Finally return the answer.

[0090] The present invention provides detailed extraction steps in the prompt of step S5 to guide the large model to complete the entire text structuring task step by step. Specifically, it clearly requires the large model to strictly follow the given steps to perform the structuring task. The steps that the large language model needs to follow are listed in the prompt.

[0091] For example, an example is as follows: "...you will extract strictly according to the user's requirements and perform the extraction according to the following steps (requirements): 1) Answer this question: What is the subject of the input text? 2) Understand the extraction requirements, what entities / objects are to be extracted? 3) Determine whether the subject is consistent with the extraction requirements, if not, give up the extraction; 4) Extract one by one according to the given entities, attributes and relationships. 5) Return the extraction results in JSON format."

[0092] S5.4, Reflection: After the above steps, the JSON formatting output capability of the large language model will be greatly improved, but there will still be errors in the format. The present invention directly calls the program to parse the JSON result output by the large language model. Specifically, in Python, the result returned by the model is directly loaded through the function json.loads. If an exception is thrown, the parsing fails, and the error is resolved through try

[0093] -catch captures the cause of the exception, appends the output of the large model and the cause of the exception to the historical conversation information through multiple rounds of dialogue, interacts with the large model again, and uses prompts to let the large language model reflect, regenerate the parsing results based on the cause of the error, and further improve the JSON output capability. The reflection prompt is "The JSON format you output after structuring is wrong, and the cause of the error is {exception}. Please restructure {input} according to the cause of the error and the requirements {schema} to ensure that the output JSON format is correct."

[0094] It can be seen from the above embodiments that fine-tuning the large model by using a weighted cross entropy loss function can force the large model to pay more attention to the generation of characters related to the JSON format syntax during text structuring, thereby improving the availability of the large model in generating JSON format strings; by presetting the identity, it is beneficial for the large model to perform targeted processing in the specified field and improve the processing effect; by using CoT to perform strict instruction requirements and extraction step guidance, the large model is prompted to perform reasoning and thinking, thereby improving the large model's content extraction effect; by emphasizing the output requirements for the JSON format in the prompt, the JSON formatted output capability of the large model is improved, and through few-shot, the extraction effect and formatted output capability of the large model are improved, and the formatted output capability can be further improved by combining the reflection strategy of the JSON parsing results.

[0095] Embodiment 2, as Figure 2 As shown, an embodiment of the present invention provides a text structuring system based on a large language model, including:

[0096] JSON format analysis module 1, used to determine the hierarchical structure, key-value pair characteristics and data type of the JSON format by analyzing the JSON format of the large language model;

[0097] The training data processing module 2 is used to perform data training based on the hierarchical structure, key-value pair characteristics and data types determined in the JSON format;

[0098] Large language model training module 3 is used to fine-tune the large language model, improve the large language model's ability to extract domains, and improve its ability to output in JSON format;

[0099] Deployment module 4, used for fine-tuning large language model deployment using vLLM;

[0100] The CoT prompt module 5 is used to use the deployed large language model to structure the text by using the CoT prompt method. For JSON format data with errors, a reflection strategy of the JSON parsing results is adopted to improve the structured output capability of the large language model.

[0101] Experimental example.

[0102] 1. The JSON format described in the present invention specifically refers to the JSON-ld format that complies with the requirements of www.schema.org. This format is a semantic web data exchange standard that mainly includes entity types, attributes, and relationships, such as:

[0103] {"Entity Type":"Athlete",

[0104] "Entity Name":"The general name of the athlete",

[0105] "Sports":"A specific activity or competition in a sport.",

[0106] "Main Award": "Honorary award for outstanding performance.",

[0107] "Graduation School":"The school the character graduated from.",

[0108] "Birthplace":"The specific place where the character was born."}

[0109] }

[0110] Among them, context is the source of the specification definition of the JSON-ld, @type indicates the type of the entity, name is the name of the entity, location indicates the place where the entity occurs, and this location is also an entity type with corresponding attribute relationships, etc.

[0111] 2. The training data processing described in the present invention mainly comes from the examples section of the website www.schema.org, which contains untagged questions and formatted text (JSON-ld), which are processed to form text pairs. In addition, the data set for entity extraction and relationship extraction comes from iepile. Following the data method of iepile, the final data set is as follows: {"instruction":"You are an expert in event extraction. Please extract events that conform to the schema definition from the input. Non-existent events return an empty list, non-existent arguments return NAN, and if the arguments have multiple values, please return a list. Please answer in the format of a JSON string.","schema":[{"event_type":"Executive changes","trigger ":true,"arguments":["Executive Name","Position after Change"]},{"event_type":"Corporate Financing","trigger":true,"arguments":["Amount of Financing","Funding Round","Event Time","Disclosure Time","Investors","Lead Investor"]}],"input":"China Railway Construction (01186.HK)'s subsidiary won the bid for the Singapore Metro Network Project with a contract value of 3.8 billion RMB\\nFeel the true temperature of the Chinese economy and witness the pace of progress in the era of chasing dreams. \\nWho can represent the strongest driving force of business in 2019? \\nClick to vote and select your "Top Ten Economic Figures of the Year in 2019". ","output":{"Executive Changes\":[],"Corporate Financing\":[]}}

[0112] 3. The training described in the present invention specifically refers to fine-tuning based on an open source large model, such as qwen2-7b-instruct as a base large model, using full parameter fine-tuning, and the relevant parameters of the fine-tuning are set as follows:

[0113] Epoch: 1;

[0114] Train_batch_size_per_gpu:1;

[0115] Max_seq_length: 4096;

[0116] Lr: 4e-5;

[0117] Loss function weight setting: JSON punctuation: 2, other symbols: 1;

[0118] The fine-tuning environment is A800*1;

[0119] The choice of base models and fine-tuning methods is unlimited.

[0120] 4. The model fine-tuned by the present invention uses vLLM to deploy the inference service. Specifically, after installing vLLM, deploy it through the following command:

[0121] python-Oum vllm.entrypoints.openai.api_server\

[0122] --host 0.0.0.0\

[0123] --port$port\

[0124] --model$MODEL_DIR\

[0125] --api-key "EMPTY"\

[0126] --tensor-parallel-size $psize\

[0127] --swap-space 0\

[0128] --disable-custom-all-reduce\

[0129] --trust-remote-code;

[0130] 5. The prompt scheme of the present invention is as follows: [

[0132] {

[0133] "role":"system",

[0134] "content":"""You are an expert who focuses on text analysis and knowledge extraction. You can accurately understand and execute the user's extraction requirements. You have strong text analysis and information extraction capabilities and can output the results in accurate JSON format. Your task is to determine whether the input text meets the extraction requirements based on the given pattern, and extract entities and attributes that match the user's requirements. The single-value attribute is a string, and multi-value attributes are output as a list.

[0135] We will extract knowledge step by step according to the following process and give the thinking process:

[0136] 1) Carefully read the text entered by the user, understand and summarize the text content;

[0137] 2) Extract knowledge based on the specific description of the attribute. When extracting, first repeat the meaning of the attribute, and then extract the attribute in the input text based on the attribute meaning.

[0138] 3) Check if the extracted attribute contains the entity name. If not, return a null value. The null value answer format is "\nThe final answer is {}".

[0139] 4) If the entity name is extracted, the entity name and other attributes are output in JSON format. The answer format is "\nThe final answer is xxx"""",

[0140] },

[0141] {

[0142] "role":"user",#, single value is a string, multiple values ​​are a list;

[0143] "content":f"""Input text: "Ye**, female, born on June 3, 1964 in a certain city in a certain province. A female speed skater from a certain country. She entered the speed skating class of Changchun Amateur Sports School at the age of 10 and was selected for the speed skating team at the age of 12. She graduated from the School of Economics and Management of a certain university in 2000 and was promoted to major general (promoted in 2006). In December 2021, the MV of "See You" sung by Ye** and 55 other world champions was released. ”,\nExtract entities of type “athlete (entity name, sports event, graduate school, birthplace)” from the input text. The attributes of the types to be extracted are explained as follows: “Entity name” is used to describe the name or title of an entity. “Sports event” refers to a specific, organized activity or competition form in the field of sports, representing sports practices participated in by different groups. “Major awards” refer to honorary awards for outstanding performance, such as trophies or medals, used to recognize athletes’ outstanding achievements in competitions. “Graduated school” refers to the school where the person graduated. “Birthplace” refers to the specific place where the person was born. The answer format is “The final answer is xxx”"""

[0144] },

[0145] {

[0146] "role":"assistant",

[0147] "content":"""1. Read and understand the input text content. The input text mainly describes the deeds of Ye**, a female speed skater from a certain country;

[0148] 6) Extract attributes according to user requirements: "Entity Name" is used to describe the name or title of an entity. The entity name is generally located at the beginning or end of the text. In this article, the only entity name related to the "athlete" is "Ye **", which is of the "athlete" type. "Ye **" will be used as the entity name.

[0149] 7) Extract attributes according to user requirements: "Sports Event" refers to specific and organized activities or competition forms in the field of sports, representing sports practices participated by different groups. The text mentions "female speed skater from a certain country", so speed skating for women from a certain country is Ye **'s sports event.

[0150] 8) Extract attributes according to user requirements: "Graduated From" refers to the school where a person graduated. The text mentions "graduated from the School of Economics and Management of a certain university in 2000", so the School of Economics and Management of a certain university is Ye **'s graduated from school.

[0151] 9) Extract attributes according to user requirements: "Place of Birth" refers to the specific place where a person was born. The text mentions "Ye **, female, born in a certain city in a certain province on June 3, 1964", so a certain city in a certain province is Ye **'s place of birth.

[0152] 10) The extracted result contains the "Entity Name" attribute, proceed to the next step;

[0153] 11) Check the extracted attributes. The extracted entity name is "Ye **", indicating that the extraction result is consistent with the user requirements. The entity name and other attributes will be output in JSON format. The final answer is {"Entity Name":"Ye **","Sports Event":"Speed Skating","Graduated From":"School of Economics and Management of a certain university","Place of Birth":"A certain city in a certain province"}

[0154] },

[0155] {

[0156] "role":"user", #, single value is a string, multiple values are a list;

[0157] "content":f"""Input text: "Ye**, female, born on June 3, 1964 in a certain province and city. A female speed skater from a certain country. In December 2021, the MV of "See You" sung by Ye** and 55 other world champions was released. ",\nExtract entities of type "Artist (entity name, agency, representative works)" from the input text. If the extracted entity does not match the required type, it will not be output; if there are multiple entities, the most important entity will be selected for output. The attributes of the types to be extracted are explained as follows: "Entity name" is used to describe the name or title of an entity, usually for various types of entities, such as people, places, organizations, products, etc. "Agency" refers to the company responsible for managing and representing artists. "Representative works" refers to the works that best reflect the person's ideological level and artistic style. The answer format is "The final answer is xxx""""

[0158] },

[0159] {

[0160] "role":"assistant",

[0161] "content":"""1. Read and understand the input text content. The input text mainly describes the deeds of Ye**, a female speed skater from a certain country;

[0162] The input text mainly describes the deeds of speed skaters, which is inconsistent with the entity type to be extracted "Artist (entity name, agency, representative work)", so the extraction is terminated. \nThe final answer is: {}; """

[0163] },

[0164] {

[0165] "role":"user",#, single value is a string, multiple values ​​are a list;

[0166] "content":f"""Input text: "Ye**, female, born on June 3, 1964 in a certain province and city. A female speed skater from a certain country. In December 2021, the MV of "See You" sung by Ye** and 55 other world champions was released. ",\nExtract entities of type "athlete (entity name, gender, project)" from the input text. If the extracted entity does not match the required type, it will not be output; if there are multiple entities, the most important entity will be selected for output. The attributes of the types to be extracted are explained as follows: "Entity name" is used to describe the name or title of an entity, usually for various types of entities, such as people, places, organizations, products, etc. "Gender" is used to identify a person's lineage and identity. "Project" refers to a specific plan or task for something. The answer format is "The final answer is xxx""""

[0167] },

[0168] {

[0169] "role":"assistant",

[0170] "content":"""1. Read and understand the input text content. The input text mainly describes the deeds of Chinese female speed skater Ye**;

[0171] 12) The Chinese female speed skater described in the input text is consistent with the entity type "athlete" to be extracted, and the next step is carried out;

[0172] 13) The attributes to be extracted are entity name, gender, and event. According to my knowledge, except for the entity name, none of them is a common attribute of the entity type "athlete". The extraction is terminated and the process goes to step 14);

[0173] 14) At step 13), the extraction is terminated and a null value should be returned. \nThe final answer is: {}""",

[0174] },

[0175] {

[0176] "role":"user",#, single value is a string, multiple values ​​are a list;

[0177] "content":f"""Input text: "Ye**, female, born on June 3, 1964 in a certain province and city. A female speed skater from a certain country. In December 2021, the MV of "See You**" sung by Ye** and 55 other world champions was released. ",\nExtract entities of type "entity (overview, company, alias)" from the input text. If the extracted entity does not match the required type, it will not be output; if there are multiple entities, the most important entity will be selected for output. The attributes of the types to be extracted are explained as follows: "Overview" is used to describe a concise summary of the basic characteristics or main points of things. "Company" is used to identify a business entity that creates, owns or manages a brand. "Alias" refers to other names of things or names in different contexts. The answer format is "The final answer is xxx""""

[0178] },

[0179] {

[0180] "role":"assistant",

[0181] "content":"""1. Read and understand the input text content. The input text mainly describes the deeds of Chinese female speed skater Ye**;

[0182] 15) The entity type to be extracted is "entity", which is a general term for all entities. It is impossible to further determine whether the extracted text meets the requirements, so the extraction is terminated and the process goes to step 16);

[0183] 16) At step 15), the extraction is terminated and a null value should be returned. \nThe final answer is: {}""",

[0184] },

[0185] {

[0186] "role":"user",#, single value is a string, multiple values ​​are a list;

[0187] "content":f"""Input text: "{input}"\nExtract entities of type "{type_str}" ({attr_list_strs}) from the input text. If the extracted entity does not match the required type, it will not be output; if there are multiple entities, the most important entity will be selected for output. The attributes of the type to be extracted are explained as follows: "{attr_descrp_strs}". The answer is output in JSON format, with key-value pairs separated by English commas, key values ​​separated by English colons, and surrounded by English double quotes. The answer format is "The final answer is xxx"""",

[0188] }

[0189] Where type_str is the entity type with extraction, and attr_list_strs is the related attribute list of the entity type with extraction.

[0190] "You are an expert focusing on text analysis and knowledge extraction,..." is the preset identity. "Key-value pairs are separated by English commas, and key values ​​are separated by English colons,..." is the output format requirement. "You will extract knowledge step by step according to the following process, and give your thinking process..." is the CoT. The second to the second to last JSON is a few-shot, which is used to guide the output of the large language model. If the JSON parsing of the answer output by the large language model fails, two JSONs will be appended after the above prompt: one to pass the output result of the large language model, and the other to pass the exception that the program failed to parse the JSON answer, prompting the large language model to restructure. The restructured prompt is as follows:

[0191] "The extraction result you provided reported an error during JSON parsing. The specific reason is {exception}. Please re-extract according to the reason. The extraction result is in JSON format and the answer format is..."

[0192] Experimental results:

[0193] The present invention was verified on a zero-sample knowledge extraction dataset provided by a university in Zhejiang Province. The F1 score of the base model (Qwen2-7B-Instruct) on this dataset was 35.24 points. After supervised fine-tuning, the F1 score was improved to 62.49 points. Combined with the prompt word engineering, the final score was 70.36 points.

[0194] Precision Recall F1-Score Qwen2-7B-Instruct 34.22 36.32 35.24 Qwen2-7B-Instruct+SFT 62.74 62.24 62.49 Qwen2-7B-Instruct+SFT+Prompting 72.12 74.34 73.21

[0195] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A text structuring method based on a large language model, characterized in that: The method includes: S1, determining the hierarchical structure, key-value pair characteristics, and data types of the JSON format by analyzing the JSON format of the large language model; S2, data training based on the hierarchical structure, key-value pair characteristics, and data types determined by the JSON format; S3, fine-tuning the large language model to improve its ability to extract domains and output JSON formatting; S4, fine-tuning of large language model deployment using vLLM; S5, using the CoT prompt method to use the deployed large language model to structure the text. For JSON format data with errors, the reflection strategy of JSON parsing results is adopted to improve the structured output capability of the large language model.

2. The text structuring method based on a large language model according to claim 1, characterized in that: In step S1, the string in JSON format is: {"aaa":"bbb","ccc":["ddd","eee"],"fff":{"ggg":"hhh"}} Among them, square brackets and curly brackets appear in pairs, strings are in double quotes, key-value pairs are separated by commas, and key values ​​are separated by colons; The large language model generates the current character according to the above context, and each character is generated by selecting the character according to probability.

3. The text structuring method based on a large language model according to claim 1, characterized in that: In step S2, data training is performed, including: crawling example rows from schema.org to form a seed set to form an information extraction dataset for a specified task; combining existing public information extraction datasets to form a multi-task learning dataset, and identifying and correcting the JSON format from different tasks.

4. The text structuring method based on a large language model according to claim 1, characterized in that: In step S3, the large language model is fine-tuned, including: using a weighted cross entropy loss function to give different weights to the prediction results of certain specific JSON punctuation characters in the text structuring task to get more attention; the weighted cross entropy loss function is expressed as: In the formula, θ is the parameter of the model, x is the input sequence; y is the actual label, that is, the next character in the sequence; w y is the weight of the previous sample y, which is the element corresponding to the current sample selected from the weight vector w; C is the character set, [y=c] is the indicator function, which takes the value of 1 when y=c, otherwise it is 0; pθ(y|x) is the probability of predicting the target y when the model is given an input x; w is the weight vector, and the elements of the weight vector correspond to the weight of each category in C.

5. The text structuring method based on a large language model according to claim 4, characterized in that: The fine-tuning includes: fine-tuning on the base model based on a weighted cross entropy loss function to improve the large language model's ability to extract domains and its ability to output JSON formatting.

6. The text structuring method based on a large language model according to claim 1, characterized in that: In step S5, the deployed large language model is used to structure the text using CoT prompts, and few-shot is used to improve the structuring effect.

7. The text structuring method based on a large language model according to claim 1, characterized in that: Step S5 specifically includes: S5.1, Identity recognition: In command editing, when interacting with the large language model, a role or identity is set for the large language model through a preset identity, and the model responds and acts according to the preset identity during the conversation or task execution; the way to preset the identity is to specify the role played by the large model through text; S5.2, Output format: In the JSON format output instructions of the large language model, the requirements of JSON format syntax and punctuation are emphasized; S5.3, using CoT prompts to enable large language models to generate intermediate reasoning steps, improve the extraction performance of large language models in complex tasks, and increase the interpretability of large language model outputs; S5.4, Reflection: Directly call the program to parse the JSON result output by the large language model. If the parsing fails, capture the cause of the exception and feed it back to the large language model, allowing the large language model to reflect and regenerate the parsing result based on the cause of the error, further improving the JSON output capability.

8. The text structuring method based on a large language model according to claim 7, characterized in that: In step S5.3, the CoT prompt includes: Problem definition: Identify the multi-step problem to be solved; Problem decomposition: Breaking down a problem into smaller tasks or problems; Create prompts: Create prompts for each small task to guide the large language model to reason; Execution: Prompts are fed to a large language model in sequence and the output of each step is collected; Analyze and optimize: Evaluate the output and optimize the prompts if necessary.

9. A text structuring system based on a large language model, characterized in that: The system implements the text structuring method based on a large language model as described in any one of claims 1 to 8, and the system includes: A JSON format analysis module (1) is used to determine the hierarchical structure, key-value pair characteristics and data type of the JSON format by analyzing the JSON format of the large language model; A training data processing module (2), used for performing data training based on the hierarchical structure, key-value pair characteristics and data types determined in the JSON format; A large language model training module (3) is used to fine-tune the large language model, improve the large language model's ability to extract domains, and improve its ability to output in JSON format; A deployment module (4), used for fine-tuning large language model deployment using vLLM; The CoT prompt module (5) is used to use the deployed large language model to structure the text by using the CoT prompt method. For JSON format data with errors, a reflection strategy of the JSON parsing result is adopted to improve the structured output capability of the large language model.

10. The text structuring system based on a large language model according to claim 9, characterized in that: The system is mounted on a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the functions of the above-mentioned text structuring system based on a large language model are realized.

Images