Method and system for automatically labeling fine tuning instruction set of large language model

Abstract

The invention discloses an automatic tagging method and system for a fine tuning instruction set of a large language model, and the method comprises the steps: obtaining to-be-tagged original corpus data; performing desensitization and cleaning preprocessing on the to-be-labeled original corpus data; based on the scene of the original corpus data, matching a corresponding pre-trained large language model to label the corpus data, and generating a labeled label field; verifying the labeled label field through a regular statement, wherein verification comprises verification of the data content and the data format of the label field; and according to the scene of the original corpus data, assembling the generated label fields on the basis of an SQL (Structured Query Language) to form a fine tuning training set of a matched large language model. According to the method, a systematic judicial data analysis and labeling process is formed by processing judicial corpus data, so that rapid analysis and labeling of judicial data in the judicial vertical field are realized, and enabling of a text processing task in the judicial field is realized by a large language model.

Description

Technical Field

[0001] The present invention relates to the technical field of instruction set annotation, and in particular to an automatic annotation method and system for fine-tuning instruction sets using a large language model. Background Art

[0002] Automatic data labeling technology refers to the automatic labeling of data through machine learning or deep learning algorithms. It is a common type of automatic data labeling, which is semi-supervised learning. Semi-supervised technology refers to the use of a small amount of labeled data and a large part of unlabeled data for data labeling, which can achieve better performance on a limited labeled data set and make full use of the information of unlabeled data.

[0003] The current automatic data annotation technology has the following technical problems: (1) The annotation results often have the defect of low accuracy, so manual correction is required when the annotation results are generated. However, manual correction requires the data correction personnel to have a certain level of professional knowledge in the data business field, and the correction results are also easily affected by the subjective cognition of the correction personnel to a great extent; (2) It is a technical challenge to align the small amount of labeled data in semi-supervised technology with the large amount of unlabeled data, which makes semi-supervised technology often suffer from insufficient labeling accuracy; For example: The invention patent with application number 202211547222 provides a technical solution for data labeling: a semi-supervised data automatic labeling method based on deep learning, the specific steps include: First, manually label some data in the dataset to be labeled, and divide the dataset to be labeled into labeled dataset and unlabeled dataset; second, automatically label the unlabeled dataset based on the BEV deep learning algorithm to obtain the labeled dataset and the new unlabeled dataset; then, determine whether the new unlabeled dataset is empty; if the new unlabeled dataset is not empty, evaluate the new labeled dataset and the new unlabeled dataset based on the binary classification deep learning algorithm; The technical solution of this patent has the following problems: (1) The effect of BEV deep learning algorithm is limited, and the algorithm may produce certain errors in the automatic labeling process; (2) The effectiveness of evaluating models based on binary classification deep learning algorithms is questionable. Not only does it require fine-tuning the model based on the initial classification results (which increases time costs and complicates the entire process, and the fine-tuning effect is affected by various factors, such as data business fields, initial classification results, and fine-tuning parameters), but it is also difficult to define the indicator parameters for evaluation using binary classification models. (3) Implementing a complete data annotation strategy often requires time and development costs to write code to implement functions such as corpus data preprocessing and fine-tuning models. If the data annotation strategy is to be modified, the code needs to be modified again. This will bring challenges such as low development efficiency and low flexibility.

[0004] In addition, compared with text processing tasks in traditional fields, text processing tasks in the legal field face challenges such as long legal documents and complex file formats. In the process of implementing text processing tasks based on prompt word engineering, it is necessary to configure the system with prompt words that strictly meet the requirements of legal documents. In the current traditional method of using large language models for text processing tasks, it is impossible to quickly parse and annotate judicial data in the legal vertical field, and enable large language models to realize text processing tasks in the judicial field.

[0005] Therefore, based on the current situation, it is urgent to improve the existing technology. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for automatically labeling a large language model fine-tuning instruction set to solve the problems raised in the above background technology.

[0007] On the one hand, the present invention provides a method for automatically labeling a large language model fine-tuning instruction set, and the specific steps include: Step S100: obtaining original corpus data to be annotated, wherein the original corpus data includes judicial corpus; Step S200: corpus data processing, desensitizing and cleaning the original corpus data to be annotated by using regular expressions and SQL statements based on the corpus processing component, wherein the cleaning preprocessing includes removing non-local regulations, regulations with non-compliant chapter and article structures, regulations with incorrect chapter titles, and regulations with mismatched regulation types from data field values; Step S300: Based on the scenario of the original corpus data, a corresponding pre-trained large language model is matched from a large language model library to annotate the original corpus data to generate an annotated label field, where the label field is the annotation result; Step S301: Access the pre-trained large language model through the intelligent pre-annotation operator, including Chatglm3-6b, Qwen1.5-14b, Qwen1.5-7b, Qwen2-72b, Shusheng Puyu, and OneKe, and select the large language model for annotation according to the text processing task and data attributes in the current original corpus data scenario to be annotated, for the text processing task of natural language processing; Step S302: Obtain prompt words through prompt word engineering based on the intelligent pre-annotation operator to guide the large language model to generate semantic annotation results under the text processing task; Step S400: verify the annotated tag field through a regular statement. If the verification result shows that it does not meet the requirements, return to step S300 to regenerate the annotation. If the verification result shows that it meets the requirements, enter step S500. The verification includes verifying the data content and data format of the tag field; the correct data format is a JSON array; Step S500: According to the scenario of the original corpus data, the generated annotated label fields are assembled based on SQL to form a fine-tuning training set of a matching large language model, wherein the fine-tuning training set includes instructions, input and output key fields, and the instructions are used to indicate the task objectives to be performed by the large language model.

[0008] On the other hand, the present invention also provides the following technical solution: a large language model fine-tuning instruction set automatic annotation system, comprising: a data reading and writing component, a corpus processing component, an intelligent pre-annotation operator and a corpus version management module; The data reading and writing component includes: a corpus input component and a corpus output component, wherein the corpus input component is used to connect the original corpus data to the corpus; the corpus output component outputs the annotation results to the corpus for storage; the corpus establishes a connection with the component library through the data reading and writing component, and the data reading and writing component, the corpus processing component, the intelligent pre-annotation operator and the corpus version management module can be directly called from the component library to create a corpus data annotation strategy; The corpus processing component includes: a corpus desensitization component and a corpus cleaning component, wherein the corpus desensitization component implements desensitization by configuring the corpus desensitization component parameters by itself to identify regular expressions for sensitive content; the corpus cleaning component cleans the field values ​​of the selected fields by using custom regular expressions and SQL statements; The intelligent pre-labeling operator implements text processing for text summarization tasks, text outline generation tasks, text classification tasks, and text entity relationship extraction tasks by accessing a large language model: The corpus version management module is used to display the annotated data with version numbers generated by various corpus annotation strategies, and provide data export options in multiple file formats.

[0009] The present invention has the following beneficial effects: (1) The present invention creates a corpus data annotation strategy based on a corpus input component, a corpus processing component, an intelligent pre-annotation operator, a corpus output component and a corpus version management module. By learning a large amount of corpus data and training on a wide range of corpora, the present invention has good semantic understanding capabilities and can directly generate high-quality annotation results without fine-tuning.

[0010] (2) Verify the annotation results through regular statements to ensure that the generated results meet the predetermined content and format requirements. This verification mechanism is equivalent to a "manual review" of the automatic annotation results, ensuring the accuracy of the annotation results in an automated manner.

[0011] (3) The entire system is modularly designed so that each link can work independently and efficiently. Users can configure annotation strategies through low-code or zero-code methods, thus surpassing the traditional manual annotation process in terms of efficiency.

[0012] (4) Through the expected version management module, users can track and manage different versions of annotation results to ensure that the annotation quality is still controlled during the automation process.

[0013] (5) In the face of the challenges of long files and complex file formats in judicial data, the present invention forms a systematic judicial data analysis and annotation process through the processing of judicial corpus data, enabling the large language model to realize text processing tasks in the judicial field. Compared with the prior art that generates results through fine-tuning of the deep learning model, the large language model used in the present invention does not need to be fine-tuned, but directly obtains the generation results through prompt word engineering based on the pre-trained model. When the generation capability needs to be adjusted, the user only needs to modify the prompt word or replace the required large language model in the platform's large language model library. This method significantly reduces the complexity of model tuning, and users can quickly realize the adaptive adjustment of the model in a short time, avoiding the tedious fine-tuning process in traditional deep learning methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the process flow of the automatic annotation method for fine-tuning instruction sets of a large language model provided by the present invention; Figure 2 This is a schematic diagram of the corpus annotation process in a judicial scenario in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of the automatic annotation system for fine-tuning instruction sets of large language models provided by the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field of the present invention without creative work are within the scope of protection of the present invention.

[0016] refer to Figure 1 On the one hand, the present invention provides a method for automatically labeling a large language model fine-tuning instruction set, the steps comprising: Step S100: raw data is accessed into the corpus, and raw corpus data to be annotated is obtained based on the corpus input component; Specifically, the original corpus data includes a variety of different formats, such as document file types and table file types. The original corpus data is imported into the corpus through the system's corpus input component. For example, in the judicial application scenario, the national laws and regulations data are imported into the corpus as original corpus data, based on the full legal and regulatory database.

[0017] Step S200: corpus data processing, desensitizing and cleaning the imported original corpus data based on the corpus processing component through regular expressions and SQL statements; wherein the corpus processing component includes: a corpus desensitization component and a corpus cleaning component; the steps include: Step S201: In the process of corpus desensitization, corpus desensitization includes five desensitization options: ID card, license plate number, address, mobile phone number and custom; among them, the ID card, license plate number, address, and mobile phone number options have built-in regular expression recognition strategies, and users can achieve desensitization without additional configuration; and the custom option achieves desensitization by custom designing or writing regular expressions to identify sensitive content.

[0018] Step S202: During the corpus cleaning process, the user selects the field to be cleaned in the corpus, and the field value of the selected field is cleaned through a custom SQL statement based on the corpus cleaning component. The cleaning processing types include removing English, invisible characters, web page identifiers, emoticons, etc. in the field values, converting traditional Chinese characters in the field values ​​into simplified Chinese characters, and normalizing spaces. As an optional embodiment of the present invention, in a judicial corpus scenario, various regulatory data in the regulatory database are cleaned, and the regulatory data are cleaned through custom SQL statements, and non-local regulations, regulations with non-compliant chapter and article structures, regulations with incorrect chapter titles, and regulations with mismatched regulatory types are screened out through field values.

[0019] As an optional embodiment of the present invention, this example mainly introduces the main actions of the present invention in the corpus processing stage; Step 1: Establish the content and format rules of the corpus data expected in the current annotation scenario, for example, determine which content in the data is sensitive and what content or format will be considered dirty data; Step 2: Develop a corpus processing strategy based on the expected data content and format rules established in the first step: In the desensitization stage, we first determine whether the existing tools for the four desensitization options of ID card, license plate number, address and mobile phone number meet the requirements: if the input original corpus data contains at least one of the desensitization options, the tools for the existing desensitization options meet the requirements; otherwise, the tools for the existing desensitization options cannot meet the requirements, and it is necessary to write corresponding regular expressions in the custom desensitization module to process the content that needs to be desensitized.

[0020] In the data cleaning stage, call the data cleaning tool to remove English characters, invisible characters, web page identifiers, and emoticons from field values, normalize spaces, and convert traditional Chinese characters into simplified Chinese characters. If the current data cleaning tool cannot meet the needs of the corpus data, you can use custom SQL statements to implement specific data cleaning functions. For example, when processing judicial corpora, determine whether the text complies with the predetermined chapter structure rules, such as checking whether each chapter contains a title and clauses, and ensuring that the numbering and format of the clauses are consistent. By executing SQL queries, identify and exclude non-compliant regulatory entries in the selected corpus data, and finally return the cleaned compliant regulatory data for subsequent analysis.

[0021] Step S300: Based on the scenario of the original corpus data, the corresponding pre-trained large language model is matched from the large language model library to annotate the original corpus data to generate an annotation result. The annotated label field specifically includes: Step S301: Access multiple large language models through intelligent pre-labeling operators, including Chatglm3-6b, Qwen1.5-14b, Qwen1.5-7b, Qwen2-72b, Shushengpu Language and OneKe, etc., to perform text processing tasks for natural language processing: Text summarization task: compressing long text into a concise summary or summary; Text outline generation task: Generate a hierarchical outline or outline to show the organization and main points of the text; Text classification task: Generate new category labels based on the original categories in the corpus data; or directly assign category labels to the text to form a labeling system; The task of extracting text entity relations: identifying entities in the text and the relations between entities.

[0022] As an optional embodiment of the present invention, refer to Figure 2 In this embodiment, taking judicial corpus as an example, the four major text processing tasks are all applied in the judicial corpus annotation scenario. For the cleaned regulations (or laws and regulations) data, the specific implementation of the four major text processing tasks is as follows: In the text summarization task, the legislative purpose generation and article title generation tasks of the regulations are realized, which include: (1) Generate legislative purpose Data extraction: In legislative purpose generation, regular expressions are first used to extract (or extract) text content from the first article of the first chapter of the regulations to identify the legislative purpose, which usually contains the legislative purpose.

[0023] Prompt word construction, for example, generating prompt words for legislative purposes in the legal field includes: Clarify the role of the big language model: ###role### You are a legal expert who is proficient in drafting local regulations; Determine the task objective of the large language model: ###task### You need to generate a sentence about the legislative purpose based on the provided legal text; The text theme is controlled by regular expressions to meet the following requirements: the generated sentences need to have clear purpose requirements; no legal names can be mentioned; the generated sentences should be around 50 words; and the legislative purpose content must not refer to other regulations.

[0024] Model call: The text extracted from the first article of the first chapter of the regulations is merged (or concatenated) with the constructed prompt words and provided as input parameters to the chatglm3-6b large language model to generate the legislative purpose.

[0025] (2) Generate the title of the regulation. The specific steps include: Data extraction: Use regular expressions to extract (or extract) the content of each article (or clause) in the regulations.

[0026] Prompt word construction, for example, generating prompt words for article titles in the legal field, including: Clarify the role of the large language model: ###role### Suppose you are an expert in summarizing the text topics based on legal texts; Determine the task objective of the large language model: ###task### You need to summarize a short text topic based on the provided legal text content; the text topic is controlled by regular expressions to meet the following requirements: no extra characters, the topic can only contain Chinese characters, English content is strictly prohibited, the topic should be less than 10 words, and only one topic is output.

[0027] Model call: The text extracted from each article is combined with the prompt word and provided as input parameters to the chatglm3-6b large language model to obtain the article title generated by the large language model.

[0028] Data processing: Each generated title is added to the original regulatory text to ensure that each article has a clear corresponding title, that is, regulations with titles for each article (or clause) are obtained. This process effectively improves the readability and structure of the regulatory text.

[0029] In this embodiment, the text outline generation task includes the following steps: Data extraction: First, the regulatory content is divided into individual chapter texts through regularization statements to facilitate outline generation; for example, a regulation with fifty chapters is divided into fifty texts, each of which contains the regulatory content of a single chapter.

[0030] Prompt word construction, for example, in the legal field, the text outline generates prompt words, including: Clarify the role of the big language model: ### role ### You act as a professional legal content summarizer; ### task ### You need to extract less than 5 short subheadings from the given content, and the number of words in each subheading should be controlled within 4-8 words; The text outline is controlled by regular expressions to meet the following requirements: the number of subheadings must be less than five; the content of each subheading must not be similar; no additional explanations can be given to the subheadings; the subheadings must be less than 10 words; the generated subheadings cannot be a declarative sentence but a general title; the generated subheadings cannot contain any English letters, English words, or English sentences; do not add any content, spaces, or line breaks before the first subheading; the generated subheadings must not contain any subheadings or other redundant supplementary content.

[0031] Model call: The extracted text is combined with the prompt words and provided as input parameters to the chatglm3-6b large language model to generate a legislative outline.

[0032] Data post-processing: Merge the legislative outlines of each chapter in order to obtain a complete outline of a regulation.

[0033] In this embodiment, the text classification task includes the following steps: Pre-category generation: In the text classification task, the title of the regulations is first analyzed by a clustering algorithm to obtain multiple topic clusters. The goal of the clustering algorithm is to divide the regulation title data into multiple clusters so that the similarity between objects in the same cluster is high, while the similarity between objects in different clusters is low. Each cluster can be regarded as a sub-region in the data space, representing a specific pattern or category. For example, when the regulation category is traffic regulations, that is, this type of regulations is laws and regulations that regulate road traffic behavior and ensure traffic safety and order. These regulations cover the behavioral standards of drivers, pedestrians, vehicles and traffic facilities, including traffic signals, speed limits, drunk driving bans, etc. The purpose of traffic regulations is to reduce traffic accidents, improve road safety, and maintain public order.

[0034] Prompt word construction: 20 title samples are extracted from each cluster to configure prompt words. In the prompt words, it is stated that the large language model is a classification expert in the legal field, and the goal of the large language model is to classify the provided regulatory titles based on the category label system, and to specify the category set generated based on the previous category in the prompt words to generate a complete legal category label system.

[0035] Model call: After merging the regulation title and prompt words, they are input into the Qwen2-72b model to generate a category label system for the regulation, that is, a set of regulation types and explanations of the types. In the process of generating the label system, the label system generated by the large language model is optimized twice through verification. The verification standard is to determine whether there are duplications in the categories and explanations generated based on the large language model to ensure that there are no duplicate categories; finally, the regulation title is spliced ​​with the updated prompt words again and input into the model to obtain the final classification result. This process ensures the systematic classification of regulation titles and improves the accuracy and efficiency of data processing.

[0036] In this embodiment, the task of extracting text entity relations includes the following steps: Data extraction: Extract chapters related to legal liability from the regulations, such as chapters related to penalties for violating the law. For example, in this embodiment, the penultimate chapter is titled Legal Liability or Penalties. A regular expression is written based on this rule to extract chapter content related to legal liability from the original regulations (or laws).

[0037] Prompt word construction: Configure prompt words, and tell the large language model in the prompt words that its role is an expert in legal entity extraction, such as: ###Your role### You are a legal expert who is proficient in summarizing the content of local regulations. Task objectives, such as: ###Your task### Your task is to strictly follow the execution steps and precautions, accurately extract the information of liability type and liability details from the provided legal regulations, and return the results strictly in JSON format.

[0038] ###Execution steps### 1. Read the legal text and for each liability type and liability details found, create a JSON object containing two key-value pairs: "liability type" and "liability details".

[0039] 2. The type of liability is criminal liability, civil liability or administrative liability. You need to determine which type of liability the legal provisions belong to based on the content of the legal provisions. The type of liability cannot be changed.

[0040] 3. The content of the responsibility details includes the punishment unit, illegal behavior, and punishment measures combined together and placed in one value. The key-value format is not allowed to appear inside.

[0041] 4. The content extracted from the penalty unit in the responsibility details includes keywords such as "department" and "staff". The extraction of the penalty unit mainly focuses on the content before the first comma of each sentence, and analyzes the subject of the sentence as the penalty unit.

[0042] 5. The content extracted from the illegal behavior in the responsibility details includes keywords such as "violation". The extraction of the penalty unit mainly focuses on the content before the first comma of each sentence, and analyzes the behavior of the subject of the sentence as the extraction content of the illegal behavior field.

[0043] 6. The content extracted from the penalty measures in the responsibility details includes keywords such as "penalty" and "order". The extraction of penalty measures mainly focuses on the content after the first comma of each sentence, analyzes the content of the sentence, and determines whether it is related to the penalty measures.

[0044] 7. If the illegal behavior in each regulation in the responsibility details consists of multiple numbers, please extract each condition in the text content of the illegal behavior and generate an independent JSON object based on each condition. The punishment method will not change according to the content of the regulation.

[0045] 8. If the regulations contain illegal acts listed in numbers, the generated JSON field of responsibility details will only contain one illegal act, and each illegal act will be included in the responsibility details, without duplication or omission.

[0046] 9. If there is no extractable information in the responsibility details, return a null value.

[0047] For the extraction of prompt words for the responsibility details entity, the input content is required to be in Chinese, and the output is a JSON array, which is wrapped by complete `[{extracted content}]`. When there is no extractable information in the text, an empty JSON array is returned; no extra characters can be added at the beginning or end of the JSON array. Use the Chinese in the original text to summarize the information, especially the responsibility type.

[0048] Prompt words for extracting entities of the legal field such as the commissioning units and the division of responsibilities: ###Your role###You are a legal expert who is proficient in summarizing the content of local regulations.

[0049] ###Your task###: Your task is to strictly follow the execution steps and precautions, accurately extract the commission and division of responsibilities information from the provided legal text, and return the results strictly in JSON format.

[0050] ###Execution steps### 1. Read the text of the law, paying special attention to the second half of the “General Provisions” section.

[0051] 2. Look for references to the commissioning units and division of responsibilities in this section.

[0052] 3. For each commission and office unit found and its division of responsibilities, create a JSON object containing two key-value pairs: "commission and office unit" and "division of responsibilities".

[0053] 4. If the description of the division of responsibilities exceeds 20 Chinese characters, summarize and generalize it to ensure that the length of the description does not exceed 20 Chinese characters.

[0054] 5. Ensure that the extracted content is pure Chinese and does not contain any Chinese punctuation, English words or letters.

[0055] 6. Group all the JSON objects into an array, which is your final output.

[0056] For the extraction of commissions and offices and division of responsibilities, when department names are involved, use the Chinese summary information in the original text.

[0057] Model call: merge the content of the legal liability chapter with the prompt words, and call the OneKe language model to identify and automatically return the corresponding entities and the relationship between entities. This process effectively extracts the key elements of legal liability.

[0058] Step S302: Generate prompt words based on the intelligent pre-labeling operator by using prompt word engineering including thought chain (a technology that studies how to write and optimize prompt words so as to better and more effectively use the large language model) to guide the large language model to generate semantic labeling results under specific text processing tasks.

[0059] As an optional embodiment of the present invention, this example mainly introduces the actions of the present invention during intelligent pre-marking or intelligent marking; Step 1: Determine the text processing task in the current annotation scenario, and decide which large language model to use for annotation based on the text processing task and data attributes. For example, if the text processing task is to extract text entity relationships, then use the oneKe large language model for annotation; if the text processing is text summarization and the data has a large number of data fields with a length of more than 8,000, then use the chatglm3-6b-32k large language model for annotation.

[0060] Step 2: Configure the large language model prompt words based on the prompt word project. The prompt words have the following key elements: A. Tell the big language model his identity, for example, tell him that he is a legal expert (depending on the specific business field); B. Tell the large language model the goal of the current generation task; C. Establish the format of the content generated by the large language model; D. List the key points of the task in an organized and logical manner, ensuring that the prompt words require the large language model to generate information in a certain logical order; E. The above ABCD steps are separated by separators; According to the prompt word, the large language model is requested to obtain the corresponding field label, that is, the annotation result, and then the regular expression statement is used to automatically verify whether the annotation result contains special characters and English. Some non-compliant data is cleaned up through the rules. For non-standard data (that is, dirty data) that cannot be cleared by the rules, the large language model is called again until the generated data meets the specifications.

[0061] Step S400: Verify the annotated tag field, i.e., the annotation result, through a regular statement to verify whether the annotation result meets the requirements, i.e., whether there are defects, such as data content, data format, etc. If the data content and format of the verified annotation result do not meet the requirements, return to the large language model to regenerate the annotation result. Exemplarily, if the regenerated annotation result still fails to pass the verification, other large language models can be reselected from the large language model library to annotate the original corpus data to be annotated, and the annotation result is re-verified after the annotation. If it meets the requirements, step S500 is executed.

[0062] Step S500: According to the currently matched large language model, the generated annotated label fields are assembled based on SQL to form a fine-tuning training set of the current large language model, that is, a complete large language model fine-tuning training set is assembled.

[0063] In this embodiment, the fine-tuning training set of the large language model is a data set prepared to enable the model to achieve better performance in specific tasks or fields. Taking the scenario of judicial corpus annotation as an example, the training set includes the fine-tuning training sets of the large language model for tasks such as legislative draft generation, legislative purpose generation, and legislative outline generation.

[0064] As an optional embodiment of the present invention, a single or multiple large language model annotated label fields have been generated before. In this example, it is necessary to assemble the label fields based on SQL according to the scenario to obtain a final large language model fine-tuning training set.

[0065] In this example, the required fields of fine-tuning training sets of different large language models are different. For example, the fine-tuning training set of the chatglm3-6b large language model has three required fields: input (input data that needs to be fine-tuned), output (expected output result), and instruction (specific instructions to the model); taking the fine-tuning task of generating legislative drafts by large language models in the judicial field as an example, the instruction is a piece of plain text, which declares that the task goal is to generate a legislative draft according to the given content (legislative title, legislative purpose, legislative basis, legislative outline and legislative type); the output content is the content of the legislative draft label; and the input field should be in the following format: "Please write an article: xxx, the legislative outline is: xxx, the purpose of the regulation is: xxx, the legislative basis is: xxx, and the type of regulation is: local regulations. Finally, the above fields are assembled based on SQL to obtain a final large language model fine-tuning training set.

[0066] Step S600: Based on the corpus version management module, the annotation result data with version numbers generated by various corpus annotation strategies are displayed, and data export options with multiple file formats are provided.

[0067] refer to Figure 3 ,On the other hand, the present invention provides the following technical solution: ,A large language model fine-tuning instruction set automatic annotation system, comprising: ,a data reading and writing component, a corpus processing component, an ,intelligent pre-annotation operator and a corpus version management module; The data reading and writing component includes: a corpus input component and a corpus output component, wherein the corpus input component is used to connect the original corpus data to the corpus; the corpus output component outputs the annotation results to the corpus for storage; the corpus establishes a connection with the component library through the data reading and writing component, and the data reading and writing component, the corpus processing component, the intelligent pre-annotation operator and the corpus version management module can be directly called from the component library to create a corpus data annotation strategy.

[0068] The corpus processing component includes: a corpus desensitization component and a corpus cleaning component, wherein the corpus desensitization component implements desensitization by self-configuring the corpus desensitization component parameters to identify regular expressions for sensitive content; the corpus cleaning component cleans the field values ​​of the selected fields through custom regular expressions and SQL statements.

[0069] The intelligent pre-labeling operator implements text processing for text summarization tasks, text outline generation tasks, text classification tasks, and text entity relationship extraction tasks by accessing a large language model: The corpus version management module is used to display the annotation result data with version numbers generated by various corpus annotation strategies, and provide data export options in multiple file formats.

[0070] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically labeling a large language model fine-tuning instruction set, characterized in that: The method comprises: Step S100: obtaining original corpus data to be annotated, wherein the original corpus data includes judicial corpus; Step S200: performing desensitization and cleaning preprocessing on the original corpus data to be annotated, wherein the cleaning preprocessing includes removing non-local regulations, regulations with non-compliant article structures, regulations with incorrect chapter titles, and regulations with mismatched regulation types in data field values; Step S300: Based on the scenario of the original corpus data, the corresponding pre-trained large language model is matched from the large language model library to annotate the original corpus data to be annotated, and a labeled label field is generated, and the label field is the annotation result; specifically comprising: step S301: accessing the pre-trained large language model, including Chatglm3-6b, Qwen1.5-14b, Qwen1.5-7b, Qwen2-72b, Shusheng Puyu, OneKe, and selecting the large language model for annotation according to the text processing task and data attributes in the current scenario of the original corpus data to be annotated; step S302: configuring the pre-trained large language model prompt words to guide the large language model to generate semantic annotation results under the text processing task; Step S400: Verify the annotated tag field. If the verification result shows that it does not meet the requirements, return to step S300 to regenerate the annotation. If the verification result shows that it meets the requirements, enter step S500. The verification includes verifying the data content and data format of the tag field. The correct data format is a JSON array. Step S500: According to the scenario of the original corpus data, the generated annotated label fields are assembled based on SQL to form a fine-tuning training set of a matching large language model.

2. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 1, characterized in that: The types of the original corpus data to be annotated in step S100 include document file types and table file types.

3. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 1, characterized in that: Step S200 also includes: Step S201: desensitizing the original corpus data, including: ID card, license plate number, address, mobile phone number and custom desensitization options; wherein, The ID card, license plate number, address, and mobile phone number options have built-in regular expression recognition strategies; The custom desensitization option processes desensitized content by customizing regular expressions.

4. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 1, characterized in that: Step S200 also includes: Step S202: corpus cleaning, selecting the fields to be cleaned in the corpus, and cleaning the field values ​​of the selected fields using custom regular expressions and SQL statements. The cleaning includes removing English, invisible characters, spaces, web page identifiers, and emoticons from the field values, and converting traditional Chinese characters in the field values ​​into simplified Chinese characters.

5. The method for automatically labeling a large language model fine-tuning instruction set according to claim 1, characterized in that: The step S302 further includes: The prompt words include four key elements: determining the role of the current large language model, the currently generated task goal, the generated content format, and the current task key points, and the key elements are separated by separators.

6. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 5, characterized in that: The text processing task includes: Text summarization task: Configure the pre-trained large language model role as a legal expert, extract text content from the regulations in the judicial corpus, merge the extracted text with the prompt word to generate the legislative purpose of the regulation, and input it as a parameter into the Chatglm3-6b large language model to generate the legislative purpose, and generate a title for each article in the regulation, and then add the generated title to the regulations in the judicial corpus to obtain regulations with a title for each article; the legislative purpose content is only extracted from the current regulations, and the generated legislative purpose does not contain any legal name.

7. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 6, characterized in that: The text processing task also includes: Text outline generation task: The regulations with titles for each article are divided into individual chapters, and then the current generation annotation tasks constructed based on the titles and prompt words generated in the text summary task in a single chapter are merged and input as parameters into the Chatglm3-6b large language model to generate a legislative outline for a single chapter, and then the legislative outlines of the single chapters are sequentially merged to obtain a complete outline; the similarity between the titles of the individual chapters is calculated one by one, and a threshold is set to absolutely exclude similar titles; Text classification task: cluster the regulations based on the generated titles to obtain multiple clusters, extract N title samples from each cluster, merge the title samples and the annotation tasks constructed by the prompt words, input them into the Qwen2-72b large language model to generate a label system for the regulations, and obtain the classification results, where N is a natural number greater than 10.

8. The automatic labeling method for fine-tuning instruction sets of a large language model according to claim 7, characterized in that: The text processing task also includes: The task of extracting text entity relationships: Identify the content of the regulatory chapter based on the label and merge it with the prompt word, call the OnkeKe large language model, automatically identify and return the entity and the relationship between entities, and output JSON array data; When there is no extractable content in the regulatory section, an empty JSON array data is returned.

9. The method for automatically labeling a large language model fine-tuning instruction set according to claim 1, characterized in that: Also includes, Step S600: Based on the corpus version management module, the generated annotation results with version numbers are displayed.

10. A large language model fine-tuning instruction set automatic annotation system, characterized in that: Using the automatic annotation method according to any one of claims 1 to 9 to create a corpus data annotation strategy and generate an annotation result; including: The data reading and writing component includes: a corpus input component and a corpus output component; wherein, The corpus input component is used to access the original corpus data into the corpus; The corpus output component is used to output the annotation results to the corpus for storage; The corpus processing component includes: a corpus desensitization component and a corpus cleaning component; wherein the corpus desensitization component configures the corpus desensitization component parameters by itself to identify regular expressions of sensitive content to achieve desensitization; the corpus cleaning component cleans the field values ​​of the selected fields by customizing regular expressions and SQL statements; Intelligent pre-labeling operator, which processes text tasks by accessing a large language model, including text summary generation, text outline generation, text classification, and extraction of text entity relationships; The corpus version management module is used to display the annotation results with version numbers generated by various corpus annotation strategies, and provide data export options in different file formats.

Images