Incremental learning-based industry large language model fine tuning method and storage medium

Abstract

The invention provides an industry large language model fine tuning method based on incremental learning and a storage medium, and the method comprises the steps: carrying out the grading labeling of collected industry data through an expert according to a grading frame preset in difficulty grade, and obtaining a training set with a hierarchical label; carrying out reinforcement learning training on the large language model according to a hierarchical sequence from easy to difficult by using the training set data to obtain a trained reinforcement large model; different learning super parameters are distributed to different decoders by analyzing training data features, incremental learning is carried out on the reinforced large model, and a trained industry large model is obtained; and collecting feedback from experts and industry users in the actual application process of the industry large model, and carrying out industry knowledge precipitation and optimization of the grading framework. The storage medium in the scheme can realize the method provided by the invention when the computer program is read and operated by the processor. According to the technical scheme, the problem of disastrous forgetting during fine adjustment of the large model is solved, and the learning of the model on domain knowledge is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence model fine-tuning, and specifically relates to a storage medium for an industry large language model fine-tuning method based on incremental learning. Background Art

[0002] In today's Natural Language Processing (NLP) field, large models are widely used. Large models are short for Large Language Models (LLMs). They have a large parameter scale and are pre-trained with large data sets. They require huge computing resources and storage space for training and storage. Large models have powerful representation capabilities and can be applied to a variety of tasks. However, when these large models are applied to specific fields, they usually need to be further fine-tuned to learn domain knowledge. This is because during pre-training, large models learn a large amount of general knowledge, which usually comes from the Internet. This type of knowledge is not effective enough in specific fields. For example, in terms of data such as national standards and related industry standards in the industrial field, if the model is not trained enough, it cannot be well applied to the gas field.

[0003] However, there are many challenges in the fine-tuning process, one of which is the catastrophic forgetting problem. Catastrophic forgetting refers to the fact that after fine-tuning a large model, the update of the neural network weights may significantly reduce some of the model's capabilities. The second key is that the professional knowledge in a specific field comes from a wide range of sources and data is diverse, so there are few high-quality and reliable training corpus samples, making it difficult to effectively fine-tune large models. Summary of the invention

[0004] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an efficient fine-tuning method for large models for specific industry fields. The method is based on expert classification, feedback, and incremental learning, and can effectively solve the problems of insufficient application capabilities of large language models in specific field scenarios, difficulty in model fine-tuning, and catastrophic forgetting in the prior art.

[0005] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0006] A method for fine-tuning a large industry language model based on incremental learning, characterized by comprising the following steps:

[0007] The collected industry data is graded and labeled by experts according to the preset grading framework of difficulty level to obtain a training set with hierarchical labels;

[0008] Performing reinforcement learning training on the large language model using the training set data in a hierarchical order from easy to difficult, to obtain a trained reinforced large model;

[0009] By analyzing the characteristics of the training data, different learning hyperparameters are assigned to different decoders, and incremental learning is performed on the enhanced large model to obtain a trained industry large model;

[0010] Collect feedback from experts and industry users during the actual application of the industry big model, accumulate industry knowledge and optimize the grading framework.

[0011] Furthermore, the preset grading framework is divided into four levels from easy to difficult, namely: basic concepts, industry regulations and company regulations, business data, and advanced theories.

[0012] Furthermore, when the training set data is used to perform reinforcement learning training on the large language model in a hierarchical order from easy to difficult, the input of the reinforcement learning training is the data source and the data text content, and the output is the level to which the data belongs.

[0013] Furthermore, the step of performing reinforcement learning training on the large language model in sequence from easy to difficult levels using the training set data to obtain a trained reinforced large model further includes iteratively training the model after reinforcement learning training:

[0014] Based on the model trained by reinforcement learning, the remaining unlabeled industry data collected is automatically evaluated at the hierarchical level;

[0015] The data of the automatic hierarchical evaluation is sampled and the sampled samples are evaluated by experts; if the experts confirm that the accuracy of the automatic hierarchical evaluation reaches a preset threshold, it is passed; otherwise, the experts newly annotate the data and add the newly annotated data to the training set to iteratively optimize the model trained by reinforcement learning.

[0016] Furthermore, the sources of collection of the industry data include, but are not limited to: encyclopedic knowledge, industry standards, product-related manuals, engineering specification documents, engineering application knowledge and troubleshooting manuals.

[0017] Furthermore, the method of allocating different learning hyper-parameters to different decoders by analyzing the characteristics of the training data includes:

[0018] Generate corresponding training parameters according to the data features in the training set, wherein the training data features include: the number and proportion of characters in data texts at different levels, the average number of characters in data texts at all levels, the total number of characters in all data sets, the word frequency of data texts at different levels, and the average character length of sentences in data texts at different levels; the training parameters include the number of batches, the initial learning rate, and the weight decay coefficient during training at different levels;

[0019] The training is organized in order from easy to difficult according to the level of the training data, and the parameters of the training process are adjusted through the training scheduler during training.

[0020] Furthermore, the training scheduler controls the learning rate to gradually decay within each level as the training progresses; when new category samples participate in the training, the learning rate will increase to the initial learning rate and then decay.

[0021] Furthermore, the method allocates different learning hyperparameters to different decoders by analyzing the characteristics of the training data, performs incremental learning on the enhanced big model, and obtains the trained industry big model, and also includes: deploying the industry big model in actual applications, and iteratively training the hierarchical capabilities of the industry big model that perform poorly based on feedback.

[0022] Furthermore, the iterative training iteratively trains the hierarchical capabilities of the industry large model that perform poorly based on feedback, including:

[0023] During actual use, experts annotate and report examples of poor performance of the industry model, and mark the level to which the corresponding problem belongs;

[0024] During the training process, the training data of the corresponding level is mainly used, and a small amount of knowledge from other levels is mixed in. New training hyperparameters are generated for the training data, and different learning hyperparameters are assigned to different decoders by analyzing the characteristics of the training data, so as to perform further incremental training on the large industry model.

[0025] The technical solution of the method of the present invention improves the model's learning of domain knowledge based on the organization form of domain training data of prior knowledge; at the same time, a special fine-tuning scheme is used to reduce the catastrophic forgetting problem during fine-tuning of large models, and expert feedback is introduced to guide the training of large models, thereby improving the application ability of large models in domain scenarios.

[0026] The present invention also provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the industry large language model fine-tuning method based on incremental learning as described above is implemented.

[0027] The advantages of the storage medium provided by the present invention and the industry large language model fine-tuning method based on incremental learning compared with the prior art are basically the same, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for fine-tuning an industry large language model based on incremental learning according to an embodiment of the present invention;

[0029] Figure 2A schematic diagram of the process of incremental learning and optimization of a reinforced large model according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the process of reinforcement learning and optimization of the initial large model according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0033] According to an embodiment of the present application, an embodiment of a method for fine-tuning a large language model for an industry based on incremental learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here. Figure 1 The method for fine-tuning an industry large language model based on incremental learning provided in an embodiment of the present application includes the following specific steps:

[0034] Step S1: The collected industry data is graded and labeled by experts according to a preset grading framework of difficulty levels to obtain a training set with hierarchical labels.

[0035] Specifically, industry data can be obtained from a variety of different sources, including encyclopedic knowledge, industry specifications, product-related manuals, engineering specification documents, engineering application guides, and troubleshooting manuals. In this embodiment, the preset grading framework sets four levels according to the difficulty of the knowledge, and the four levels are arranged from easy to difficult:

[0036] 1. Basic concepts: This is the most basic level, including basic definitions, basic principles, and basic concepts in the field. This part of knowledge does not involve complex tasks such as reasoning and judgment. The content is relatively simple and is the basis of other levels of knowledge. The model understands basic concepts by learning this part of knowledge, so this part of knowledge is the first part to be trained.

[0037] 2. Industry regulations and company regulations: This level includes laws and regulations, industry standards, industry specifications and company regulations in the field, such as engineering safety regulations, standard regulations for engineering equipment and company safety regulations. This part of knowledge will have simple judgment, reasoning and other types of tasks.

[0038] 3. Business data: This part of the data is generated in the actual business process, including industry deployment cases, internal equipment installation and various operation logs. This part of the data will have more complex logical reasoning and calculation tasks.

[0039] 4. Advanced theory: This part of the data includes advanced principles in the field, such as the working principles of engineering equipment, the innovative principles of engineering equipment, etc.

[0040] In particular, a portion of the collected industry data can be left without expert-level labeling and used for iterative optimization of the adjusted model.

[0041] Step S2: Perform reinforcement learning training on the large language model using the training set data in order from easy to difficult levels to obtain a trained reinforced large model.

[0042] Specifically, as mentioned above, the training set can be divided into basic concept training set, industry regulations and company regulations training set, business data training set, and advanced theoretical data set according to the level. In the intensive training, the initial large language model is trained in the order of basic concept training set, industry regulations and company regulations training set, business data training set, and advanced theoretical data set, where the input of the large language model is the data source and data text content, and the output is the level to which it belongs. Through this hierarchical data organization method, the data is organized according to the difficulty of the knowledge it contains, which is conducive to the model learning industry knowledge from easy to difficult. In particular, the data is split into four levels in this scheme, which can actually be split into several levels according to actual industry knowledge; the data and model can also be split at different levels, which can be split and combined according to actual experience.

[0043] This step S2 also includes iterative training of the model after reinforcement learning training, as follows:

[0044] Based on the model trained by reinforcement learning, the remaining unlabeled industry data collected is automatically evaluated at the hierarchical level;

[0045] 10% of the data of automatic hierarchical evaluation is sampled and the sampled samples are evaluated by experts; if the experts confirm that the accuracy of automatic hierarchical evaluation reaches the preset threshold of 80%, it is passed; otherwise, the experts newly annotate the data and add the newly annotated data to the training set to iteratively optimize the model trained by the reinforcement learning. Of course, the sampling ratio and threshold can be adjusted according to the actual situation.

[0046] In this embodiment, the knowledge level is divided according to the industry prior knowledge, and the data is classified according to the divided levels. In this way, the training data set can be organized in the order of easy first and difficult later, so that the model can learn basic knowledge first and then learn complex business logic and other task contents during the training process. At the same time, the solution introduces reinforcement learning, combined with feedback from industry experts, so that the model can continuously optimize the data stratification during the training generation process and improve the training data organization effect.

[0047] Step S3: By analyzing the characteristics of the training data, different learning hyperparameters are assigned to different decoders, and incremental learning is performed on the enhanced large model to obtain the trained industry large model. Specifically, it includes:

[0048] S31: Generate relevant training parameters based on the characteristics of the training data. The characteristics of the training data include: the number and proportion of data at different levels, the average number of characters, the total number of characters, the word frequency, and the average length of each sentence. The relevant training parameters include: Since data at different levels are trained in sequence during training, the training hyperparameters of data at different levels are different. The training hyperparameters for training data at different levels include the number of batches, the initial learning rate, the weight decay coefficient, etc. When the fine-tuning method is LoRA, the hyperparameter alpha is also added as a generation parameter.

[0049] In particular, the number of characters referred to in this embodiment refers to the number of words after word segmentation after the original data text is segmented, for example, "Today's weather is good." After word segmentation, it can be divided into three words: "Today", "Weather" and "Good", that is, the number of characters after word segmentation in this sentence is 3. The average number of characters refers to the average number of all levels, the total number of characters refers to the total number of characters in all training data sets, and the number and proportion of data at different levels also refer to the statistical number and proportion of the number of characters. The average length of a sentence refers to the average character length of sentences of data at different levels. The word frequency calculation refers to the calculation of the top 500 most frequently occurring words and their number of occurrences based on statistical analysis of original text data at different levels.

[0050] S32: The training data organizes the four types of data in order from easy to difficult, and during training, the training scheduler is used to adjust some parameters in the model training process. The training scheduler controls the learning rate to gradually decay within each category as the training process progresses. When new category samples participate in the training, the learning rate will increase to the initial learning rate and then decay. And different initial learning rates can be set for each category sample.

[0051] Specifically, take the batch number parameter as an example: Assuming the benchmark is 32 (the benchmark needs to be verified in different scenarios), adjust it according to the data characteristics of the level to which the training data belongs. If the current level data ratio is higher than the average ratio, increase the parameter by the corresponding ratio. If the average length of the sentences in the current level data exceeds the average length of the data in all levels, reduce the parameter by the corresponding ratio. Round the calculated parameter as the final parameter, where the batch number needs to be a multiple of 2.

[0052] The learning rate is adjusted in the opposite way to the batch size parameter, that is, if the current level data accounts for a high proportion, the parameter needs to be reduced proportionally. However, no rounding is required, and the range is 3e-5 to 3e-4.

[0053] The weight decay coefficient is adjusted in the same way as the batch number parameter, that is, if the current level data accounts for a high proportion, this parameter needs to be increased, ranging from 0 to 1e-5.

[0054] The training hyperparameters are only related to the level to which the training data belongs.

[0055] S33: Deploy the application model in the actual environment. During use, experts will annotate and feedback the samples where the large model performs poorly and mark the level of the corresponding problem.

[0056] S34: Further incremental training is performed for the level capabilities of the model with poor performance to enhance the knowledge of the corresponding level of the model. During the training process, the corresponding level training data is mainly used, and a small amount (less than 30%) of other level knowledge is mixed in. And new training hyperparameters are generated for the training data, that is, steps S31 and S32 are repeated.

[0057] The structure of a large language model is usually composed of multiple fully connected decoders. The lower and higher layers play different roles in solving tasks, and it can be considered that the large model capabilities corresponding to different layers are different. Therefore, for industry data, this embodiment allocates different learning hyperparameters to different decoders by analyzing the characteristics of training data, ensuring that new industry knowledge can be learned while reducing the loss of original capabilities.

[0058] Step S4: Collect feedback from experts and industry users during the actual application process to accumulate industry knowledge and optimize the grading framework. After the large industry model is trained, it is deployed to the actual application scenario. In this process, feedback from experts and industry users is collected to understand the performance of the model in the actual environment and improve the model.

[0059] In summary, this technical solution first introduces expert knowledge in the process of organizing data and model training, and designs expert feedback content in combination with a unique data stratification scheme. The introduction of expert feedback effectively improves the quality of training data organization and the effectiveness of incremental model training, which is conducive to the model's better improvement of industry knowledge capabilities. Secondly, a new large model training method is proposed, including hierarchical data organization and training hyperparameter generation based on hierarchical data. Through hierarchical data organization, data is organized according to the difficulty of the knowledge it contains, which is conducive to the model learning industry knowledge in order from easy to difficult; by combining the characteristics of hierarchical data for hyperparameter generation, the model can better learn the training data, while balancing the model's learning of new knowledge and retention of old knowledge, thereby effectively avoiding the occurrence of catastrophic forgetting. Finally, expert feedback is introduced in both training data organization and model training iteration to ensure the effectiveness of model output and improve the application capabilities of large models in domain scenarios.

[0060] Another embodiment of the present invention provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the industry large language model fine-tuning method as described above is implemented.

[0061] The technical effect that can be achieved by this embodiment is basically the same as the technical effect that can be achieved by the aforementioned industry large language model fine-tuning method, and will not be repeated here.

[0062] An electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein.

[0063] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0064] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0065] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0066] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for fine-tuning a large industry language model based on incremental learning, characterized by The steps include: The collected industry data is graded and labeled by experts according to the preset grading framework of difficulty level to obtain a training set with hierarchical labels; Performing reinforcement learning training on the large language model using the training set data in a hierarchical order from easy to difficult, to obtain a trained reinforced large model; By analyzing the characteristics of the training data, different learning hyperparameters are assigned to different decoders, and incremental learning is performed on the enhanced large model to obtain a trained industry large model; Collect feedback from experts and industry users during the actual application of the industry big model, accumulate industry knowledge and optimize the grading framework.

2. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 1, characterized in that The preset grading framework is divided into four levels from easy to difficult, namely: basic concepts, industry regulations and company regulations, business data, and advanced theories.

3. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 1, characterized in that When the training set data is used to perform reinforcement learning training on the large language model in a hierarchical order from easy to difficult, the input of the reinforcement learning training is the data source and the data text content, and the output is the level to which the data belongs.

4. A method for fine-tuning a large industry language model based on incremental learning as described in any one of claims 1 to 3, characterized in that The method further comprises performing reinforcement learning training on the large language model in sequence of the training set data from easy to difficult levels to obtain a trained reinforced large model, and iteratively training the model after reinforcement learning training: Based on the model trained by reinforcement learning, the remaining unlabeled industry data collected is automatically evaluated at the hierarchical level; The data of the automatic hierarchical evaluation is sampled and the sampled samples are evaluated by experts; if the experts confirm that the accuracy of the automatic hierarchical evaluation reaches a preset threshold, it is passed; otherwise, the experts newly annotate the data and add the newly annotated data to the training set to iteratively optimize the model trained by reinforcement learning.

5. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 1, characterized in that The sources of collection of the industry data include but are not limited to: encyclopedia knowledge, industry standards, product-related manuals, engineering specification documents, engineering application knowledge and troubleshooting manuals.

6. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 1, characterized in that The method of allocating different learning hyper-parameters to different decoders by analyzing the characteristics of the training data includes: Generate corresponding training parameters according to the data features in the training set, wherein the training data features include: the number and proportion of characters in data texts at different levels, the average number of characters in data texts at all levels, the total number of characters in all data sets, the word frequency of data texts at different levels, and the average character length of sentences in data texts at different levels; the training parameters include the number of batches, the initial learning rate, and the weight decay coefficient during training at different levels; The training is organized from easy to difficult according to the level of the training data, and the parameters of the training process are adjusted through the training scheduler during training.

7. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 6, characterized in that The training scheduler controls the learning rate to gradually decay within each layer as the training progresses; when new category samples participate in the training, the learning rate will increase to the initial learning rate and then decay.

8. A method for fine-tuning a large industry language model based on incremental learning as described in any one of claims 1, 6 or 7, characterized in that The method of allocating different learning hyperparameters to different decoders by analyzing the characteristics of training data, performing incremental learning on the enhanced big model, and obtaining the trained industry big model, also includes: deploying the industry big model in actual applications, and iteratively training the hierarchical capabilities of the industry big model that perform poorly based on feedback.

9. The method for fine-tuning a large industry language model based on incremental learning as claimed in claim 8, characterized in that The iterative training iteratively trains the hierarchical capabilities of the industry large model that perform poorly based on feedback, including: During actual use, experts annotate and report examples of poor performance of the industry model, and mark the level to which the corresponding problem belongs; During the training process, the training data of the corresponding level is mainly used, and a small amount of knowledge from other levels is mixed in. New training hyperparameters are generated for the training data, and different learning hyperparameters are assigned to different decoders by analyzing the characteristics of the training data, so as to perform further incremental training on the large industry model.

10. A computer-readable storage medium, characterized in that: It includes a computer-readable storage medium storing a computer program and a processor. When the computer program is read and executed by the processor, the method for fine-tuning an industry large language model based on incremental learning as described in any one of claims 1 to 9 is implemented.

Images