Instruction data construction method and system for fine tuning of large language model

Abstract

The invention provides an instruction data construction method and system for fine tuning of a large language model. The instruction data construction method comprises the following steps: collecting semi-structured data and non-structured data; cleaning the semi-structured data to obtain a first instruction data set; cleaning the unstructured data, and segmenting the cleaned unstructured data into text blocks; inputting the text block into a preset large language model, and generating a second instruction data set through the large language model; inputting the first instruction data set and the second instruction data set into a large language model, and generating a third instruction data set through the large language model; and screening the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set. According to the method, large-scale high-quality instruction data can be quickly constructed, fine tuning of a large language model is effectively supported, and the labor cost is saved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a method and system for constructing instruction data for fine-tuning a large language model. Background Art

[0002] The training process of a large language model is generally divided into two stages: pre-training and fine-tuning (instruction fine-tuning). In the pre-training stage, the language model is pre-trained through unlabeled text data, and in the fine-tuning stage, the model parameters are further adjusted through supervised instruction data to adapt it to specific tasks. However, high-quality instruction data often requires a lot of manpower to label, and the cost and complexity of data labeling are high, which limits the fine-tuning effect of large language models.

[0003] To address the above problems, current solutions include Self-Instruct and Self-QA, which automatically build instruction datasets based on GPT-4. However, these methods have high requirements for data cleaning and preparation, the quality of the generated data is difficult to guarantee, and it is difficult to cover all possible tasks and scenarios.

[0004] Therefore, how to ensure the diversity and high quality of instruction data while reducing labor costs, thereby improving the instruction-following ability of large language models, is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a method and system for constructing instruction data for fine-tuning a large language model, so as to address the defects in the prior art of high requirements for data preparation, difficulty in ensuring the quality of generated data, and difficulty in covering all possible tasks and scenarios, thereby realizing the automatic construction of a large-scale high-quality instruction data set, improving the instruction-following effect of a large language model, and saving labor costs.

[0006] The present invention provides a method for constructing instruction data for fine-tuning a large language model, comprising the following steps.

[0007] Collect semi-structured and unstructured data; Cleaning the semi-structured data to obtain a first instruction data set; Cleaning the unstructured data, and dividing the cleaned unstructured data into text blocks; Inputting the text block into a preset large language model, and generating a second instruction data set through the large language model; Inputting the first instruction data set and the second instruction data set into the large language model, and generating a third instruction data set through the large language model; The first instruction data set, the second instruction data set and the third instruction data set are screened to obtain a fourth instruction data set.

[0008] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, each of the semi-structured data includes two parts: a question and an answer, and each of the unstructured data consists of a text paragraph, and each of the text paragraphs consists of free text without predefined association relationships.

[0009] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of cleaning the semi-structured data specifically includes: S31, preprocessing the semi-structured data to remove data with a text length that is too short or has too many illegal characters in the semi-structured data; S32. Use BERT to calculate the text perplexity of each semi-structured data after preprocessing; S33. Remove semi-structured data whose text perplexity is higher than a preset perplexity first threshold.

[0010] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of cleaning the unstructured data specifically includes: S41, preprocessing the unstructured data to remove data with a text length that is too short or has too many illegal characters in the unstructured data; S42, removing personal privacy data from the pre-processed unstructured data; S43. Use BERT to calculate the text perplexity of each unstructured data. S44. Remove unstructured data whose text perplexity is higher than the preset perplexity second threshold.

[0011] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of dividing the cleaned unstructured data into text blocks specifically includes: Each cleaned unstructured data is divided into paragraphs.

[0012] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of inputting the text block into a preset large language model and generating a second instruction data set through the large language model specifically includes: S61, inputting the text block into a preset large language model, and generating a question corresponding to the text block through the preset large language model; S62: Based on the question and the text block, generate an answer corresponding to the question through a preset large language model; S63. Concatenate the questions and the answers corresponding to the questions to form a first question-answer pair, and combine the first question-answer pairs into a second instruction data set.

[0013] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of generating a third instruction data set through the large language model based on the first instruction data set and the second instruction data set specifically includes: S71, combining the first instruction data set and the second instruction data set into a seed instruction set; S72, randomly extracting a number of seed instructions from the seed instruction set, and generating questions and answers from the seed instructions through a preset large language model to form a second question-answer pair; S73, determining the scores of each of the newly generated second question-answer pairs by using a preset large language model, and screening each of the second question-answer pairs based on a preset score threshold to remove second question-answer pairs with scores lower than the preset score threshold; S74, expanding the screened second question-answer pairs into the seed instruction set; S75. Repeat S72 to S74 until the preset number of seed instruction expansion times is reached.

[0014] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the step of screening the first instruction data set, the second instruction data set, and the third instruction data set to obtain a fourth instruction data set specifically includes: S81, converting instruction data in the first instruction data set, the second instruction data set, and the third instruction data set into an embedding vector using an embedding model; S82, calculating the cosine similarity between the embedding vectors; S83, removing instruction data corresponding to the embedded vectors whose cosine similarity is higher than a preset similarity threshold.

[0015] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the embedding model is one of a BERT, Sentence Transformers and Word2Vec model.

[0016] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, after obtaining the fourth instruction data set, the method further includes: Based on the fourth instruction dataset, fine-tune the target large language model.

[0017] According to the instruction data construction method for fine-tuning a large language model provided by the present invention, the Lora algorithm is used to fine-tune the target large language model; The Lora algorithm adjusts specific layers of a pre-trained target large language model by inserting a learnable low-rank adaptation matrix through low-rank decomposition.

[0018] The present invention also provides an instruction data construction system for fine-tuning a large language model, comprising the following modules: A collection module for collecting semi-structured and unstructured data; A first instruction data set acquisition module, used for cleaning the semi-structured data to obtain a first instruction data set; A text segmentation module, used for cleaning the unstructured data and segmenting the cleaned unstructured data into text blocks; A second instruction data set acquisition module, used for inputting the text block into a preset large language model, and generating a second instruction data set through the large language model; A third instruction data set acquisition module, configured to input the first instruction data set and the second instruction data set into the large language model, and generate a third instruction data set through the large language model; The fourth instruction data set acquisition module is used to screen the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set.

[0019] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the instruction data construction method for fine-tuning a large language model as described in any one of the above is implemented.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the instruction data construction method for fine-tuning a large language model as described in any one of the above.

[0021] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-described instruction data construction methods for fine-tuning a large language model.

[0022] The instruction data construction method and system for fine-tuning a large language model provided by the present invention collects semi-structured data and unstructured data; cleans the semi-structured data to obtain a first instruction data set; cleans the unstructured data and divides the cleaned unstructured data into text blocks; inputs the text blocks into a preset large language model, and generates a second instruction data set through the large language model; inputs the first instruction data set and the second instruction data set into the large language model, and generates a third instruction data set through the large language model; and screens the first instruction data set, the second instruction data set, and the third instruction data set to obtain a fourth instruction data set. Compared with the prior art, this solution can quickly construct large-scale, high-quality instruction data with less manpower, and effectively support the fine-tuning of a large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flowchart of the instruction data construction method for fine-tuning a large language model provided by the present invention.

[0025] Figure 2 It is one of the structural schematic diagrams of the instruction data construction system for fine-tuning a large language model provided by the present invention.

[0026] Figure 3 This is the second structural schematic diagram of the instruction data construction system for fine-tuning a large language model provided by the present invention.

[0027] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention is described in detail below in conjunction with the accompanying drawings of the specification. The specific operating method in the method embodiment can also be applied to the device embodiment or the system embodiment. In the description of the present invention, unless otherwise specified, "at least one" includes one or more. "Multiple" refers to two or more. For example, at least one of A, B and C includes: A exists alone, B exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B and C exist at the same time. In the present invention, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a kind of association relationship that describes the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0030] The present invention will be described in detail below in conjunction with specific implementation modes.

[0031] In some specific embodiments of the present invention, Figure 1 As shown, this solution provides a method for constructing instruction data for fine-tuning a large language model, including: Step 100: Collect semi-structured data and unstructured data; Step 200: clean the semi-structured data to obtain a first instruction data set; Step 300: clean the unstructured data, and divide the cleaned unstructured data into text blocks; Step 400: input the text block into a preset large language model, and generate a second instruction data set through the large language model; Step 500: input the first instruction data set and the second instruction data set into the large language model, and generate a third instruction data set through the large language model; Step 600: Screen the first instruction data set, the second instruction data set, and the third instruction data set to obtain a fourth instruction data set.

[0032] It should be noted that the existing instruction data construction solutions have high requirements for data cleaning and preparation, the quality of the generated data is difficult to guarantee, and it is difficult to cover all possible tasks and scenarios.

[0033] Therefore, the present invention integrates semi-structured data and unstructured data, uses a preset powerful language model to generate diverse instruction data, and screens high-quality data to fine-tune the large language model. It can automatically build a large-scale, high-quality instruction data set while reducing manual participation, thereby significantly improving the instruction-following effect of the large language model.

[0034] The above steps are described in detail below through specific embodiments.

[0035] Step 100: Collect semi-structured data and unstructured data; In some possible implementations of the present invention, each piece of the semi-structured data includes two parts: a question and an answer, and each piece of the unstructured data consists of a text paragraph, and each of the text paragraphs consists of free text without a predefined association relationship.

[0036] Specifically, this embodiment provides an implementation method for semi-structured data and unstructured data. The semi-structured data consists of two parts: questions and answers, and the unstructured data consists of text paragraphs. Through the comprehensive collection of semi-structured data and unstructured data, a rich corpus source is provided for the pre-training of the target large language model to be fine-tuned and the construction of an instruction data set.

[0037] In a possible embodiment, the semi-structured data includes question-answering data of a publicly accessible Internet question-answering platform, and the unstructured data includes pages of Wikipedia.

[0038] Specifically, first, semi-structured data and unstructured data are comprehensively collected. Semi-structured data mainly includes question-and-answer data collected from publicly accessible Internet question-and-answer platforms, which usually have a relatively clear structure, that is, each piece of data consists of a question part and a corresponding answer part. The advantage of this data structure is that it not only retains the user's real question-and-answer method, but also contains direct answer information, so it is very suitable for training the question-and-answer ability of the language model in actual scenarios. To collect these question-and-answer data, this embodiment selects multiple mainstream Internet question-and-answer platforms, such as Quora, Stack Overflow, Zhihu, etc. These platforms cover a wide range of topics and user questions, including but not limited to fields such as science and technology, medicine, law, education, and daily life. By crawling and filtering the data of these platforms, a large number of representative questions and answer pairs can be obtained, providing a basis for subsequent data processing and model training. In terms of unstructured data, all pages from Wikipedia are mainly collected. Wikipedia covers all kinds of knowledge from natural sciences to humanities and social sciences, with the characteristics of rich content, timely updates, and wide coverage. Unstructured data usually exists in the form of continuous text paragraphs without strict structure or labels. These text paragraphs include various content elements such as article titles, text, references, and external links. By fully crawling Wikipedia pages, a large amount of free text data can be obtained, providing a rich source of corpus for the pre-training of large language models and the construction of instruction datasets.

[0039] Step 200: clean the semi-structured data to obtain a first instruction data set; In some possible implementations of the present invention, the step of cleaning the semi-structured data specifically includes: S31, preprocessing the semi-structured data to remove data with a text length that is too short or has too many illegal characters in the semi-structured data; S32. Use BERT to calculate the text perplexity of each semi-structured data after preprocessing; S33. Remove semi-structured data whose text perplexity is higher than a preset perplexity first threshold.

[0040] Specifically, this embodiment provides an implementation method for cleaning semi-structured data, in which the semi-structured data is cleaned by a rule-based and semantic method to obtain cleaned semi-structured data.

[0041] In a possible embodiment, when removing data with too short text length, a minimum text length threshold is first set to ensure that each piece of data contains enough information to make it of practical value for the training of the language model. For question and answer data, the number of characters of the question and answer is required to reach a certain length to ensure that they can express clear semantics and complete content. For example, the minimum length of the question part is set to 10 characters, and the minimum length of the answer part is set to 20 characters.

[0042] In a possible embodiment, when removing data with too many illegal characters, natural language processing tools (such as spell checkers, grammar checkers) are used to automatically detect grammatical errors and spelling errors in question and answer data. Minor errors can be corrected by automatic correction tools; question and answer pairs with too many errors or that cannot be automatically corrected are removed from the data set to ensure data quality. Then, illegal characters (such as non-standard language symbols, non-ASCII characters, etc.) are defined, and the presence of these characters is detected by writing regular expressions or other character filtering methods.

[0043] In a possible embodiment, BERT is used to calculate the text perplexity of each piece of data, specifically including: inputting each piece of data into the BERT model and calculating its perplexity score. A lower perplexity indicates that the text is more similar to the corpus trained on the model, and is easier to understand and generate; while a higher perplexity indicates that the text is more difficult to understand and may contain noise, ambiguity, or uncommon structures. The text with a lower perplexity is retained.

[0044] Furthermore, a first perplexity threshold is preset, and data with a text perplexity higher than the set first perplexity threshold is removed. Specifically, after calculating the text perplexity for each structured data, a pre-set perplexity threshold is used to filter the data. If the perplexity score of a piece of data is higher than the set threshold (e.g., 50), it is considered as data that is not suitable for model training and is deleted from the data set.

[0045] Step 300: clean the unstructured data, and divide the cleaned unstructured data into text blocks; In some possible implementations of the present invention, the step of cleaning the unstructured data specifically includes: S41, preprocessing the unstructured data to remove data with a text length that is too short or has too many illegal characters in the unstructured data; S42, removing personal privacy data from the pre-processed unstructured data; S43. Use BERT to calculate the text perplexity of each unstructured data. S44. Remove unstructured data whose text perplexity is higher than the preset perplexity second threshold.

[0046] Specifically, this embodiment provides an implementation method for cleaning the unstructured data, wherein the unstructured data is cleaned by a rule-based and semantic method to obtain cleaned unstructured data.

[0047] In a possible embodiment, when removing data with too short text length, a minimum text length threshold is first set to ensure that each piece of data contains enough information to make it of practical value for the training of the language model. For question and answer data, the number of characters of the question and answer is required to reach a certain length to ensure that they can express clear semantics and complete content. For example, the minimum length of the question part is set to 10 characters, and the minimum length of the answer part is set to 20 characters.

[0048] In a possible embodiment, when removing data with too many illegal characters, illegal characters (including non-standard language symbols and non-ASCII characters) are defined, and the existence of these characters is detected by writing regular expressions and other character filtering methods.

[0049] In a possible embodiment, when removing personal privacy data, entries that may contain personal privacy information are identified and removed. Personal privacy data includes name, ID number, phone number, email address, home address, and other information that may be used to identify an individual. Named entity recognition technology is used to automatically detect personal information in text through a pre-trained model. After identifying this sensitive information, choose to delete it from the text or replace it with a common term (such as "someone" or "somewhere"). This step is designed to protect user privacy, ensure compliance with data use, and reduce potential legal and ethical risks.

[0050] In a possible embodiment, BERT is used to calculate the text perplexity of each piece of data, specifically including: Each piece of data is fed into the BERT model and its perplexity score is calculated. A lower perplexity means that the text is more similar to the corpus trained on the model and is easier to understand and generate; a higher perplexity means that the text is more difficult to understand and may contain noise, ambiguity, or uncommon structures. The text with the lower perplexity is retained.

[0051] Furthermore, a second perplexity threshold is preset to remove data whose text perplexity is higher than the set second perplexity threshold. Specifically, after calculating the text perplexity for each structured data, a pre-set perplexity threshold is used to filter the data. Specifically, if the perplexity score of a piece of data is higher than the set threshold (set to 50 in this example), it is considered as data that is not suitable for model training and is deleted from the data set.

[0052] In some possible implementations of the present invention, the step of dividing the cleaned unstructured data into text blocks specifically includes: Each cleaned unstructured data is divided into paragraphs.

[0053] Specifically, the cleaned unstructured data is segmented into paragraphs and divided into multiple text blocks to avoid the length of each data being too long to meet the maximum context length requirement of the preset large language model such as GPT-4.

[0054] Step 400: input the text block into a preset large language model, and generate a second instruction data set through the large language model; In some possible implementations of the present invention, the step of inputting the text block into a preset large language model and generating a second instruction data set through the large language model specifically includes: S61, inputting the text block into a preset large language model, and generating a question corresponding to the text block through the preset large language model; S62: Based on the question and the text block, generate an answer corresponding to the question through a preset large language model; S63. Concatenate the questions and the answers corresponding to the questions to form a first question-answer pair, and combine the first question-answer pairs into a second instruction data set.

[0055] Specifically, this embodiment provides an implementation method for generating a second instruction data set through a preset large language model, by inputting the segmented text blocks into the preset large language model, generating question-answer pairs through the large language model, and the generated question-answer pairs form a second instruction data set.

[0056] In a possible embodiment, the preset large language model can select a strong language model, such as a GPT-4 model. Based on GPT-4, the text block is used to generate a second instruction data set.

[0057] Specifically, the second instruction dataset is generated by using the GPT-4 model. Specifically, the cleaned structured data is divided into multiple text blocks, and GPT-4 is used to generate questions and answers based on these text blocks. The generated second instruction dataset contains new question-answer pairs that reflect the key information in the original text blocks and are expressed in natural language. In this way, a high-quality instruction dataset covering a wide range of content can be constructed for training and fine-tuning large language models to enhance their ability to understand and answer various practical problems.

[0058] In a possible embodiment, GPT-4 is used to generate questions based on the text blocks, and for each text block, a GPT-4 model is used to generate relevant questions. Each text block is input into GPT-4, and GPT-4 automatically generates a question based on the content, subject, and contextual information of the text block. The generated questions are intended to reflect the main concepts, facts, or information points in the text block, so that the model can extract key information from the text. When generating questions, GPT-4 takes into account the question format, semantic clarity, and contextual relevance of natural language to ensure that the generated questions are reasonable, easy to understand, and can accurately match the content of the text block. This step provides a targeted basis for subsequent answer generation.

[0059] In a possible embodiment, GPT-4 is used to generate answers based on the questions generated in step S301 and the corresponding text blocks of the responses. In this step, GPT-4 is used to generate answers based on the questions generated in step S301 and the corresponding text blocks. Each generated question is input into the GPT-4 model together with the original text block. GPT-4 automatically generates answers corresponding to the questions by analyzing the semantics of the questions and the content in the text blocks. The generated answers are based on the information in the text blocks, directly answering the questions and providing relevant details. This step ensures that each question has a suitable and accurate answer, thus forming a valid question-answer pair. The generated answer content should be logical and informative, able to fully express the information contained in the original text block, and maintain consistency with the question.

[0060] In a possible embodiment, the question and the corresponding answer are spliced ​​to form a question-answer pair. In this step, each previously generated question is spliced ​​with its corresponding answer to form a complete question-answer pair. The construction of the question-answer pair helps to convert the information in the text block into instruction data that can be directly used by the natural language processing model. By systematically organizing the question and answer pairs to form a second instruction data set with a clear structure, a standardized input data format is provided for subsequent large language model training. These question-answer pairs reflect the information in the original text block. After being generated and screened by GPT-4, the high quality and relevance of the question-answer pairs are ensured, thereby enhancing the effectiveness of the instruction data set.

[0061] Step 500: input the first instruction data set and the second instruction data set into the large language model, and generate a third instruction data set through the large language model; In some possible implementations of the present invention, the step of generating a third instruction data set by using the large language model based on the first instruction data set and the second instruction data set specifically includes: S71, combining the first instruction data set and the second instruction data set into a seed instruction set; S72, randomly extracting a number of seed instructions from the seed instruction set, and generating questions and answers from the seed instructions through a preset large language model to form a second question-answer pair; S73, determining the scores of each of the newly generated second question-answer pairs by using a preset large language model, and screening each of the second question-answer pairs based on a preset score threshold to remove second question-answer pairs with scores lower than the preset score threshold; S74, expanding the screened second question-answer pairs into the seed instruction set; S75. Repeat S72 to S74 until the preset number of seed instruction expansion times is reached.

[0062] Specifically, this embodiment provides an implementation method for generating a third instruction data set through a large language model, and by expanding the first instruction data set and the second instruction data set, large-scale high-quality instruction data can be quickly obtained.

[0063] In a possible embodiment, a process of generating a third instruction data set based on the first instruction data set and the second instruction data set using a preset large language model such as GPT-4 is used. In this embodiment, the third instruction data set is generated by using the GPT-4 model. Specifically, the third instruction data set is expanded and generated based on the first instruction data set and the second instruction data set obtained above. This process aims to further enrich the content of the instruction data set and improve the model's ability to respond to diverse user questions. The GPT-4 model uses its powerful language generation and understanding capabilities to generate new question and answer pairs by analyzing and synthesizing existing instruction data. These newly generated question and answer pairs will constitute the third instruction data set, expanding the coverage and depth of the original data set.

[0064] In a possible embodiment, the first instruction data set and the second instruction data set are used as seed instructions. Specifically, in this step, the first instruction data set and the second instruction data set that have been cleaned and obtained are merged as the initial seed instruction set. The first instruction data set comes from the cleaning result of the semi-structured data, and the second instruction data set is the instruction data generated by GPT-4 based on the structured data block. These seed instruction sets provide basic content for subsequent instruction generation, and their content covers a variety of topics and question and answer types. Merging these two data sets can improve the diversity and representativeness of instruction generation, and lay the foundation for generating a high-quality third instruction data set.

[0065] In a possible embodiment, a number of seed instructions are randomly extracted from the seed instruction set, and the seed instructions are used to generate questions and answers through a preset large language model to form a second question-answer pair process. For example, five seed instructions can be randomly extracted, and GPT-4 is used to generate questions and answers based on the five seed instructions to construct a question-answer pair. Specifically, in this step, five instructions are randomly extracted from the merged seed instruction set. Then, the GPT-4 model is used to use these extracted seed instructions as input to generate new questions and answers based on the five seed instructions to construct a new question-answer pair. When generating new question-answer pairs, GPT-4 generates questions and answers that are similar or related to the five seed instructions in terms of topic and style by analyzing the content, grammatical structure, and semantic relationship of the five seed instructions. These newly generated question-answer pairs are intended to expand the coverage of instruction data, so that the data set contains a wider range of scenarios and question types, and increase the diversity and practical application value of instruction data.

[0066] In a possible embodiment, the process of removing the second question-answer pair with a score lower than the preset score threshold, such as using GPT-4 to determine the score of the newly generated question-answer pair, removes the question-answer pair with a score lower than the set threshold. Specifically, in order to ensure the quality of the newly generated question-answer pairs, in this step, GPT-4 is used to score each newly generated question-answer pair. The scoring criteria may include multiple dimensions such as semantic relevance, grammatical correctness, accuracy of the answer, and information completeness. GPT-4 generates a comprehensive score for each question-answer pair and compares the score with a preset threshold. Question-answer pairs with scores below the threshold are deemed to be of substandard quality and will be removed from the data set. This process ensures that only high-quality content is retained in the expanded third instruction data set, avoiding low-quality or irrelevant content from affecting the model training effect.

[0067] In a possible embodiment, the process of expanding the screened second question-answer pairs to the seed instruction set is to expand the seed instructions with the question-answer pairs that have not been removed. Specifically, after scoring and screening, all question-answer pairs that have not been removed and meet the quality standards will be added to the seed instruction set. This operation not only increases the number of seed instructions, but also enriches the content diversity of the instruction set. After each expansion, the types of question-answer pairs and the scope of topics included in the seed instruction set will be improved, providing more reference samples for subsequent instruction generation. This step-by-step expansion method helps to accumulate high-quality instruction data and provide a solid foundation for further generation processes.

[0068] Furthermore, the above process can be repeated until the set number of seed instruction expansions is reached. In other words, instruction data expansion is an iterative process, and the operations of the above steps are continuously performed. Each time, five new seed instructions are randomly selected from the expanded seed instruction set, and new question-answer pairs are generated through GPT-4, which are scored and screened, and qualified question-answer pairs are added to the seed instruction set. Repeat this process until the preset number of seed instruction expansions is reached (set to 10,000 times in this example). Through repeated iterations, the third instruction data set gradually grows into a high-quality instruction data set with wide coverage and stable quality. This data set will be used to further optimize and fine-tune the large language model so that it can handle more complex and diverse user needs in practical applications.

[0069] Step 600: Screen the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set; In some possible implementations of the present invention, the step of screening the first instruction data set, the second instruction data set, and the third instruction data set to obtain a fourth instruction data set specifically includes: S81, converting instruction data in the first instruction data set, the second instruction data set, and the third instruction data set into an embedding vector using an embedding model; S82, calculating the cosine similarity between the embedding vectors; S83, removing instruction data corresponding to the embedded vectors whose cosine similarity is higher than a preset similarity threshold.

[0070] Specifically, this embodiment provides an implementation method for obtaining a fourth instruction data set, by filtering the obtained instruction data set, and forming the filtered instruction data into a fourth instruction data set.

[0071] Specifically, the embedding model can be used to filter the first instruction data set, the second instruction data set, and the third instruction data set to obtain a fourth instruction data set. In this embodiment, the first instruction data set, the second instruction data set, and the third instruction data set are filtered by using the embedding model to construct a fourth instruction data set. The purpose of the screening is to ensure the diversity and high quality of the instruction data set, reduce redundant information, and improve the effectiveness of the instruction set when training a large language model. The specific process includes converting the instruction data into an embedding vector, calculating the cosine similarity between the instruction data, and removing instruction data with too high similarity. The final fourth instruction data set will contain semantically unique and information-rich question-answer pairs, providing high-quality data input for the training of the large language model.

[0072] In a possible embodiment, in the process of converting instruction data into an embedding vector using an embedding model, each instruction data in the first instruction data set, the second instruction data set, and the third instruction data set is converted into an embedding vector using an embedding model. The embedding model uses Sentence-BERT to map natural language text to a high-dimensional continuous vector space. Each question-answer pair (including questions and answers) is used as input, and a corresponding embedding vector is generated through the embedding model. This vector representation retains the semantic information of the original text, so that the similarity between different texts can be measured by geometric distance in the vector space. This step provides a basis for subsequent similarity calculations and screening of redundant data.

[0073] In a possible embodiment, the embedding model is one of BERT, Sentence Transformers, and Word2Vec models.

[0074] In a possible embodiment, the cosine similarity between each of the embedded vectors is calculated, that is, the cosine similarity between the instruction data is calculated. In this step, the cosine similarity between each pair of instruction data is calculated based on the generated embedded vector. Cosine similarity is a measurement method to measure the degree of similarity between the directions of two vectors, usually used for text similarity analysis, and its value is between -1 and 1. By calculating the cosine similarity between embedded vectors, the semantic similarity of different instruction data can be determined. The specific implementation method is to form the embedded vectors into a matrix, and use the cosine similarity formula to calculate the similarity between each pair of vectors in the matrix. High cosine similarity indicates that the corresponding instruction data are highly similar in semantics, while low similarity indicates that the semantic difference is large. This step aims to identify semantically redundant instruction data and provide a basis for the next step of screening.

[0075] In a possible embodiment, instruction data corresponding to the embedded vectors whose cosine similarity is higher than a preset similarity threshold can be removed, and instruction data whose cosine similarity is higher than the set threshold can be removed. In this step, a similarity threshold (set to 0.8 in this example) is set according to the calculated cosine similarity to filter instruction data. If the cosine similarity of a pair of instruction data exceeds the set threshold, it is considered that one of the instruction data is redundant and can be removed. The specific implementation includes traversing the cosine similarity matrix, identifying instruction pairs whose similarity exceeds the threshold, marking one of the instruction data as redundant and deleting it from the data set. In this way, the redundant information in the instruction data set can be effectively reduced to ensure that the final fourth instruction data set only contains semantically independent and information-rich question-answer pairs. This process helps to improve the training quality and efficiency of large language models and avoid the degradation of model performance due to redundant data.

[0076] In some possible implementations of the present invention, after obtaining the fourth instruction data set, the target large language model is fine-tuned based on the fourth instruction data set.

[0077] In this step, the target large language model is fine-tuned by using the fourth instruction dataset. The purpose of fine-tuning is to further optimize the large language model so that it can answer user questions more accurately and effectively and better adapt to specific task requirements. The fourth instruction dataset contains high-quality question-answer pairs that have been screened and optimized. These question-answer pairs cover a wide range of topics and complexities, ensuring that the model can learn rich semantic information and diverse language structures during the fine-tuning process. The fine-tuning process adjusts the weights of the model so that the model can generate more accurate, relevant, and meaningful answers when dealing with questions similar to the training data.

[0078] In some possible implementations of the present invention, the Lora algorithm is used to fine-tune the target large language model; The Lora algorithm adjusts specific layers of a pre-trained target large language model by inserting a learnable low-rank adaptation matrix through low-rank decomposition.

[0079] In this embodiment, the specific fine-tuning process adopts a technology called Lora (Low-Rank Adaptation). The Lora method achieves fine-tuning by inserting a learnable low-rank adaptation matrix into a specific layer of the model. This method not only effectively reduces the computing resources required in the fine-tuning process, but also significantly reduces memory usage while maintaining the overall performance and expressiveness of the model. Lora's low-rank decomposition mechanism allows the optimization of the model to be completed by adjusting only a small number of parameters, which makes the fine-tuning process more efficient and more adaptable.

[0080] The instruction data construction method for fine-tuning a large language model provided in an embodiment of the present invention integrates semi-structured data (such as Internet question-and-answer platforms) and unstructured data (such as Wikipedia pages), uses GPT-4 to generate diversified instruction data, and selects high-quality data through an embedded model, which is finally used to fine-tune the large language model. Compared with the prior art, this method can automatically construct a large-scale, high-quality instruction data set with less manpower and reduced human participation, effectively support the fine-tuning of a large language model, and significantly improve the instruction-following effect of a large language model.

[0081] In some specific embodiments of the present invention, Figure 2 As shown, this solution provides an instruction data construction system for fine-tuning a large language model, including the following modules: A collection module 21, used for collecting semi-structured data and unstructured data; A first instruction data set acquisition module 22, configured to clean the semi-structured data to obtain a first instruction data set; A text segmentation module 23, used for cleaning the unstructured data and segmenting the cleaned unstructured data into text blocks; A second instruction data set acquisition module 24, configured to input the text block into a preset large language model, and generate a second instruction data set through the large language model; A third instruction data set acquisition module 25, configured to input the first instruction data set and the second instruction data set into the large language model, and generate a third instruction data set through the large language model; The fourth instruction data set acquisition module 26 is used to filter the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set.

[0082] In some possible implementations of the present invention, the instruction data construction system for fine-tuning a large language model further includes: a fine-tuning module 27, configured to fine-tune the target large language model based on the fourth instruction data set.

[0083] The implementation principle and beneficial effects of the instruction data construction system for fine-tuning a large language model provided in an embodiment of the present invention are similar to the implementation principle and beneficial effects of the instruction data construction method for fine-tuning a large language model shown in the above embodiment. Please refer to the implementation principle and beneficial effects of the instruction data construction method for fine-tuning a large language model shown in the above embodiment, and no further details will be given here.

[0084] In some possible embodiments of the present invention, Figure 3As shown in the figure, the instruction data construction system for fine-tuning a large language model can be divided into different functional units. Through the collaborative work of carefully designed specific modules, efficient training and optimization of the large language model in vertical fields can be achieved. The system can specifically include: A data collection module 31, used to collect semi-structured data and unstructured data, wherein the semi-structured data includes question-and-answer data of publicly accessible Internet question-and-answer platforms, and the unstructured data includes all pages of Wikipedia; A data cleaning module 32, used for cleaning the semi-structured data and the unstructured data, wherein the cleaning method of the semi-structured data includes removing data with too short text length, removing data with too many illegal characters, and removing data with text perplexity higher than a set threshold, and the cleaning method of the unstructured data includes removing data with too short text length, removing data with too many illegal characters, removing personal privacy data, and removing data with text perplexity higher than a set threshold, and the obtained data is the first instruction data set; The data generation module 33 is used to generate larger-scale instruction data based on the instruction data constructed by the aforementioned modules, and to divide the cleaned structured data into text blocks, wherein the text blocks refer to the paragraphs obtained by dividing each data; to generate a second instruction data set based on the text blocks using GPT-4, wherein the second instruction data set specifically uses GPT-4 to generate questions based on the text blocks, uses GPT-4 to generate answers based on the generated questions and the corresponding text blocks, and splices the questions and the corresponding answers to form a question-answer pair; uses GPT-4 to generate a third instruction data set based on the first instruction data set and the second instruction data set, wherein the third instruction data set specifically combines the first instruction data set and the second instruction data set. The instruction data set is used as a seed instruction, five seed instructions are randomly selected, GPT-4 is used to generate questions and answers based on the five seed instructions, question-answer pairs are constructed, GPT-4 is used to determine the scores of the newly generated question-answer pairs, question-answer pairs with scores lower than the set threshold are removed, and the question-answer pairs that are not removed are expanded into the seed instructions, and this process is repeated until the set number of seed instruction expansions is reached; the first instruction data set, the second instruction data set, and the third instruction data set are screened using the embedding model to obtain a fourth instruction data set, wherein the fourth instruction data set specifically uses the embedding model to convert the instruction data into an embedding vector, calculates the cosine similarity between the instruction data, and removes the instruction data whose cosine similarity is higher than the set threshold; The model training module 34 is used to fine-tune the target large language model using the instruction data set, wherein the fine-tuning method is specifically to adjust the specific layers of the pre-trained language model by inserting a learnable low-rank adaptation matrix through low-rank decomposition.

[0085] Through the above settings of this embodiment, the instruction data construction works together through multiple specially designed modules, and the large language model is efficiently trained and optimized through high-quality instruction data sets, thereby improving the overall performance and application value of the model. The functions of each module are closely connected, and together they constitute a complete instruction data construction and model optimization framework.

[0086] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the instruction data construction method for fine-tuning the large language model, the method comprising: collecting semi-structured data and unstructured data; cleaning the semi-structured data to obtain a first instruction data set; cleaning the unstructured data and dividing the cleaned unstructured data into text blocks; inputting the text blocks into a preset large language model, and generating a second instruction data set through the large language model; inputting the first instruction data set and the second instruction data set into the large language model, and generating a third instruction data set through the large language model; screening the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set.

[0087] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the instruction data construction method for fine-tuning a large language model provided by the above methods, the method including: collecting semi-structured data and unstructured data; cleaning the semi-structured data to obtain a first instruction data set; cleaning the unstructured data and dividing the cleaned unstructured data into text blocks; inputting the text blocks into a preset large language model, and generating a second instruction data set through the large language model; inputting the first instruction data set and the second instruction data set into the large language model, and generating a third instruction data set through the large language model; screening the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set; based on the fourth instruction data set.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the instruction data construction method for fine-tuning a large language model provided by the above-mentioned methods, the method comprising: collecting semi-structured data and unstructured data; cleaning the semi-structured data to obtain a first instruction data set; cleaning the unstructured data and dividing the cleaned unstructured data into text blocks; inputting the text blocks into a preset large language model, and generating a second instruction data set through the large language model; inputting the first instruction data set and the second instruction data set into the large language model, and generating a third instruction data set through the large language model; screening the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set; based on the fourth instruction data set.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing instruction data for fine-tuning a large language model, characterized in that: include: Collect semi-structured and unstructured data; Cleaning the semi-structured data to obtain a first instruction data set; Cleaning the unstructured data, and dividing the cleaned unstructured data into text blocks; Inputting the text block into a preset large language model, and generating a second instruction data set through the large language model; Inputting the first instruction data set and the second instruction data set into the large language model, and generating a third instruction data set through the large language model; The first instruction data set, the second instruction data set and the third instruction data set are screened to obtain a fourth instruction data set.

2. The method for constructing instruction data for fine-tuning a large language model according to claim 1, characterized in that: Each piece of the semi-structured data includes two parts: a question and an answer. Each piece of the unstructured data consists of a text paragraph, and each text paragraph consists of free text without a predefined association relationship.

3. The method for constructing instruction data for fine-tuning a large language model according to claim 2, characterized in that: The step of cleaning the semi-structured data specifically includes: S31, preprocessing the semi-structured data to remove data with a text length that is too short or has too many illegal characters in the semi-structured data; S32. Use BERT to calculate the text perplexity of each semi-structured data after preprocessing; S33. Remove semi-structured data whose text perplexity is higher than a preset perplexity first threshold.

4. The method for constructing instruction data for fine-tuning a large language model according to claim 2, characterized in that: The step of cleaning the unstructured data specifically includes: S41, preprocessing the unstructured data to remove data with a text length that is too short or has too many illegal characters in the unstructured data; S42, removing personal privacy data from the pre-processed unstructured data; S43. Use BERT to calculate the text perplexity of each unstructured data. S44. Remove unstructured data whose text perplexity is higher than the preset perplexity second threshold.

5. The method for constructing instruction data for fine-tuning a large language model according to claim 4, characterized in that: The steps of dividing the cleaned unstructured data into text blocks include: Each cleaned unstructured data is divided into paragraphs.

6. The method for constructing instruction data for fine-tuning a large language model according to claim 5, characterized in that: The step of inputting the text block into a preset large language model and generating a second instruction data set through the large language model specifically includes: S61, inputting the text block into a preset large language model, and generating a question corresponding to the text block through the preset large language model; S62: Based on the question and the text block, generate an answer corresponding to the question through a preset large language model; S63. Concatenate the questions and the answers corresponding to the questions to form a first question-answer pair, and combine the first question-answer pairs into a second instruction data set.

7. The method for constructing instruction data for fine-tuning a large language model according to claim 6, characterized in that: The step of generating a third instruction data set by using the large language model based on the first instruction data set and the second instruction data set specifically includes: S71, combining the first instruction data set and the second instruction data set into a seed instruction set; S72, randomly extracting a number of seed instructions from the seed instruction set, and generating questions and answers from the seed instructions through a preset large language model to form a second question-answer pair; S73, determining the scores of each of the newly generated second question-answer pairs by using a preset large language model, and screening each of the second question-answer pairs based on a preset score threshold to remove second question-answer pairs with scores lower than the preset score threshold; S74, expanding the screened second question-answer pairs into the seed instruction set; S75. Repeat S72 to S74 until the preset number of seed instruction expansion times is reached.

8. The method for constructing instruction data for fine-tuning a large language model according to claim 7, characterized in that: The step of screening the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set specifically includes: S81, converting instruction data in the first instruction data set, the second instruction data set, and the third instruction data set into an embedding vector using an embedding model; S82, calculating the cosine similarity between the embedding vectors; S83, removing instruction data corresponding to the embedded vectors whose cosine similarity is higher than a preset similarity threshold.

9. The method for constructing instruction data for fine-tuning a large language model according to claim 8, characterized in that: The embedding model is one of BERT, Sentence Transformers and Word2Vec models.

10. The method for constructing instruction data for fine-tuning a large language model according to any one of claims 1 to 9, characterized in that: After obtaining the fourth instruction data set, the method further includes: Based on the fourth instruction dataset, fine-tune the target large language model.

11. The method for constructing instruction data for fine-tuning a large language model according to claim 10, characterized in that: Use the Lora algorithm to fine-tune the target large language model; The Lora algorithm adjusts specific layers of a pre-trained target large language model by inserting a learnable low-rank adaptation matrix through low-rank decomposition.

12. A system for constructing instruction data for fine-tuning a large language model, characterized in that: include: A collection module for collecting semi-structured and unstructured data; A first instruction data set acquisition module, used for cleaning the semi-structured data to obtain a first instruction data set; A text segmentation module, used for cleaning the unstructured data and segmenting the cleaned unstructured data into text blocks; A second instruction data set acquisition module, used for inputting the text block into a preset large language model, and generating a second instruction data set through the large language model; A third instruction data set acquisition module, configured to input the first instruction data set and the second instruction data set into the large language model, and generate a third instruction data set through the large language model; The fourth instruction data set acquisition module is used to screen the first instruction data set, the second instruction data set and the third instruction data set to obtain a fourth instruction data set.

Images