Multi-stage pre-training method of professional field text large model and related device
By performing multi-stage pre-training on text big models and using cross-entropy loss for data difficulty classification and phased training, the problem of convergence in pre-training of text big models is solved, and the testing effect and robustness of the model are improved.
Patent Information
- Application Number
- CN202411034052.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-12-03
AI Technical Summary
During the pre-training process of existing text big models, the model training convergence is unstable due to the wide variety of data and different difficulty levels, and it is impossible to effectively conduct centralized training for difficult scenarios, resulting in poor test results and robustness.
Using multi-stage pre-training method, by sampling the full data in batches, calculating cross-entropy loss for difficulty classification, obtaining simple, medium and difficult-level pre-training data sets, and performing phased pre-training from easy to difficult, using Incremental learning method is used to train the model.
It improves the test results and robustness of the model, speeds up the training convergence speed, reduces the subjective intervention and labor costs of human experts, prevents knowledge forgetting, and improves the training stability of the model.
Smart Images

Figure CN119067233A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine learning, and relates to the field of large text model pre-training, and in particular to a multi-stage pre-training method and related devices for large text models in professional fields. Background Art
[0002] The training process of a large text model generally includes three stages. Explanatorily, the first stage uses a pre-training method to use various types of text data to enable the large text model to acquire basic general knowledge. The second stage uses a fine-tuning method to use data from specific scenarios to enable the large text model to acquire corresponding capabilities. The third stage uses a reinforcement learning method based on human feedback to align the output of the large text model with human cognition.
[0003] At present, the existing pre-training process of large text models usually adopts the method of completely mixed random sampling. In the pre-training process of large text models in professional fields, there is a problem of unstable model training convergence due to the wide variety of data and different levels of difficulty. Further explainably, the large text model has different levels of understanding of different data sets in different sub-fields of different professional fields (for example, for difficult scenarios, the model is more difficult to understand; for simple scenarios, the model is easier to understand). In the pre-training method of completely mixed random sampling, it is impossible to conduct concentrated training for difficult scenarios and the model training process is not stable enough, resulting in poor test results and robustness of the pre-trained large text model.
[0004] In order to make the model converge better during the pre-training process, some researchers have proposed multi-stage pre-training methods and curriculum learning methods. Both methods are not suitable for pre-training of large text models, and there is an urgent need to further develop new pre-training solutions for large text models. Specifically, the paper "Multi-stage Pre-training over Simplified Multimodal Pre-training Models" proposed a multi-stage pre-training method for multimodal pre-training models, which used three-stage pre-training from word, phrase to sentence level; however, the input and output of the large text model are complete sentences, and multi-stage pre-training cannot be performed at the word and phrase level. Therefore, the multi-stage pre-training method of the multimodal pre-training model mentioned above is not suitable for pre-training of large text models. In addition, the paper "An Asynchronous UpdatingReinforcement Learning Framework for Task-oriented Dialog System" uses a curriculum learning method to train a dialogue state tracking module. It divides user intent into three levels: simple, medium, and difficult according to the dialogue state tracking module's understanding accuracy of each user intent. However, the output of a large text model is often text, and the difficulty of the data cannot be evaluated by a simple accuracy rate. Therefore, the above-mentioned curriculum learning method is not suitable for pre-training of large text models. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-stage pre-training method and related devices for a large text model in a professional field to solve one or more of the above-mentioned technical problems. The present invention specifically discloses a multi-stage pre-training method. Compared with the existing traditional pre-training method, the large text model trained by the technical solution of the present invention has better test results and better model robustness.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a multi-stage pre-training method for a large model of professional field text, comprising the following steps:
[0008] Based on the acquired full amount of data, a pre-trained text model is trained for a preset number of rounds using a mixed random sampling method to obtain a trained text model.
[0009] The full amount of data is sampled in batches, and the cross entropy loss corresponding to each batch of sampled data is calculated based on the large text model after one training, and the difficulty of the full amount of data is classified based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level;
[0010] Based on pre-training data sets of various preset difficulty levels, the large text model after one training is pre-trained in stages from easy to difficult, and a large text model in a professional field is obtained after multi-stage pre-training.
[0011] A further preferred improvement of the present invention is that the steps of sampling the full amount of data in batches, calculating the cross entropy loss corresponding to each batch of sampled data based on a large text model trained once, and classifying the full amount of data in difficulty based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level include:
[0012] The full amount of data is sampled in batches to obtain multiple batches of sampled data; wherein, in the process of sampling in batches, for each data set in the full amount of data, the data in the data set is sampled in sequence in a sequential sampling manner until the number of sampled data reaches a preset sampling threshold, thereby obtaining a batch of sampled data;
[0013] Input each batch of sampled data into the trained text model, and calculate the cross entropy loss corresponding to each batch of sampled data;
[0014] Based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, each batch of sampled data is classified by difficulty to obtain a pre-trained data set of each preset difficulty level after difficulty classification.
[0015] A further preferred improvement of the present invention is that
[0016] In the step of classifying the difficulty of each batch of sampled data based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, and obtaining a pre-trained data set of each preset difficulty level after difficulty classification,
[0017] The preset classification thresholds are a and b, and a<b;
[0018] When classifying the difficulty of each batch of sampled data, if the cross entropy loss is less than a, the corresponding batch of sampled data is at a simple level; if the cross entropy loss is greater than or equal to a and less than b, the corresponding batch of sampled data is at a medium level; if the cross entropy loss is greater than or equal to b, the corresponding batch of sampled data is at a difficult level;
[0019] The batches of sampling data belonging to the simple level are merged, the batches of sampling data belonging to the medium level are merged, and the batches of sampling data belonging to the difficult level are merged to obtain pre-training data sets of each preset difficulty level after difficulty classification.
[0020] A further preferred improvement of the present invention is that
[0021] The steps of performing stage-by-stage pre-training from easy to difficult on the text model after one training based on the pre-training data sets of each preset difficulty level to obtain the professional field text model after multi-stage pre-training include:
[0022] Using pre-trained data sets of various preset difficulty levels from easy to difficult, a mixed random sampling method is used to train the large text model after the first training to obtain the large text model after the second training;
[0023] An incremental learning method is adopted to utilize pre-trained data sets of various preset difficulty levels and combinations of pre-trained data sets of various preset difficulty levels from easy to difficult, and a mixed random sampling method is adopted to train the large text model after secondary training to obtain a large text model in a professional field after multi-stage pre-training.
[0024] A further preferred improvement of the present invention is that
[0025] In the step of training the pre-trained text model for a preset number of rounds by adopting a mixed random sampling method based on the acquired full amount of data to obtain a trained text model,
[0026] The number of the preset rounds is at least 1 round.
[0027] In a second aspect, the present invention provides a multi-stage pre-training system for a large model of professional field text, comprising:
[0028] A one-time training module is used to train the pre-trained text model for a preset number of rounds using a mixed random sampling method based on the acquired full amount of data to obtain a one-time trained text model;
[0029] A difficulty classification module is used to sample the full amount of data in batches, and calculate the cross entropy loss corresponding to each batch of sampled data based on the large text model after one training, and classify the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level;
[0030] The secondary training module is used to perform pre-training on the large text model after the first training in stages from easy to difficult based on the pre-training data sets of various preset difficulty levels, so as to obtain a large text model in a professional field after multi-stage pre-training.
[0031] A further preferred improvement of the present invention is that
[0032] In the difficulty classification module, the steps of sampling the entire data in batches, calculating the cross entropy loss corresponding to each batch of sampled data based on the trained large text model, and classifying the entire data in difficulty based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level include:
[0033] The full amount of data is sampled in batches to obtain multiple batches of sampled data; wherein, in the process of sampling in batches, for each data set in the full amount of data, the data in the data set is sampled in sequence in a sequential sampling manner until the number of sampled data reaches a preset sampling threshold, thereby obtaining a batch of sampled data;
[0034] Input each batch of sampled data into the trained text model, and calculate the cross entropy loss corresponding to each batch of sampled data;
[0035] Based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, each batch of sampled data is classified by difficulty to obtain a pre-trained data set of each preset difficulty level after difficulty classification.
[0036] A further preferred improvement of the present invention is that
[0037] In the step of classifying the difficulty of each batch of sampled data based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, and obtaining a pre-trained data set of each preset difficulty level after difficulty classification,
[0038] The preset classification thresholds are a and b, and a<b;
[0039] When classifying the difficulty of each batch of sampled data, if the cross entropy loss is less than a, the corresponding batch of sampled data is at a simple level; if the cross entropy loss is greater than or equal to a and less than b, the corresponding batch of sampled data is at a medium level; if the cross entropy loss is greater than or equal to b, the corresponding batch of sampled data is at a difficult level;
[0040] The batches of sampling data belonging to the simple level are merged, the batches of sampling data belonging to the medium level are merged, and the batches of sampling data belonging to the difficult level are merged to obtain pre-training data sets of each preset difficulty level after difficulty classification.
[0041] A further preferred improvement of the present invention is that
[0042] In the secondary training module, the steps of executing pre-training data sets based on various preset difficulty levels, performing stage-by-stage pre-training from easy to difficult on the large text model after the primary training, and obtaining the large professional field text model after multi-stage pre-training include:
[0043] Using pre-trained data sets of various preset difficulty levels from easy to difficult, a mixed random sampling method is used to train the large text model after the first training to obtain the large text model after the second training;
[0044] An incremental learning method is adopted to utilize pre-trained data sets of various preset difficulty levels and combinations of pre-trained data sets of various preset difficulty levels from easy to difficult, and a mixed random sampling method is adopted to train the large text model after secondary training to obtain a large text model in a professional field after multi-stage pre-training.
[0045] A further preferred improvement of the present invention is that
[0046] In the one-time training module, a step is performed in which a pre-trained text model is trained for a preset number of rounds by adopting a mixed random sampling method based on the acquired full amount of data to obtain a trained text model.
[0047] The number of the preset rounds is at least 1 round.
[0048] According to a third aspect of the present invention, there is provided an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-stage pre-training method for a large professional field text model as described in any one of the first aspects of the present invention is implemented.
[0049] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the multi-stage pre-training method for a large professional field text model as described in any one of the first aspects of the present invention is implemented.
[0050] Compared with the prior art, the present invention has the following beneficial effects:
[0051] In view of the technical difficulties of unstable model training convergence caused by a wide variety of data and different levels of difficulty in the pre-training process of the professional field text large model in the prior art, the present invention specifically discloses a new multi-stage pre-training method for the professional field text large model. The professional field text large model after multi-stage pre-training of the present invention has better test results and better model robustness than the model obtained by the existing method. Specifically explanatory, in the technical solution disclosed by the present invention, the full amount of data is first sampled in batches, and then the difficulty of the full amount of data is classified based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level, and finally the pre-trained data sets of each preset difficulty level are used to train the text large model in stages, so as to obtain a text large model with good test results and robustness. To further explain, the technical solution of the present invention first samples the full amount of data in batches, and then classifies the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data. Compared with classifying the difficulty of each data set in the full amount of data as a whole, more refined classification results can be obtained, which is conducive to accelerating the convergence speed of model training; compared with manual classification, it can eliminate the gap between the feelings of human experts and the capabilities of the model itself, reduce the subjective intervention of human experts, and reduce the investment in human costs.
[0052] In a further preferred embodiment of the present invention, a training process of preview and incremental learning review is adopted during training. Compared with the traditional method of training in turns, it can enhance the learning of the basic capabilities of the model, gradually add data of higher difficulty levels, improve the training stability of the model, and effectively prevent the model from forgetting knowledge, which is conducive to training a large text model with better test results. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0054] Figure 1 It is a flowchart of a multi-stage pre-training method for a large professional field text model in an embodiment of the present invention.
[0055] Figure 2 It is a schematic diagram of the difficulty classification process in a specific embodiment of the present invention.
[0056] Figure 3 It is a schematic diagram of sample distribution of different difficulty levels of different data sets in a specific embodiment of the present invention.
[0057] Figure 4 It is a schematic diagram of a training process based on pre-training data sets of different difficulty levels in a specific embodiment of the present invention.
[0058] Figure 5 It is a schematic diagram of a multi-stage pre-training system for a large professional field text model in an embodiment of the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.
[0060] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0061] See also Figure 1 In an embodiment of the present invention, a multi-stage pre-training method for a large model of professional field text is provided, comprising the following steps:
[0062] Step 1: Based on the acquired full amount of data, a pre-trained text model is trained for a preset number of rounds using a mixed random sampling method to obtain a trained text model;
[0063] Step 2, sampling the full amount of data in batches, and calculating the cross entropy loss corresponding to each batch of sampled data based on the large text model after one training obtained in step 1, and classifying the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level;
[0064] Step 3, based on the pre-training data sets of each preset difficulty level obtained in step 2, the text big model after one training is pre-trained in stages from easy to difficult, to obtain the professional field text big model after multi-stage pre-training.
[0065] Explanatoryally, in order to solve the technical problem of unstable model training convergence caused by a wide variety of data and different levels of difficulty in the pre-training process of a large professional field text model, an embodiment of the present invention specifically provides a multi-stage pre-training method for a large professional field text model. The pre-trained large professional field text model has better test results and better model robustness than the model obtained by the existing technical solution; explanatoryly by specific example, in the training process of the large electric power text model, the large text model obtained by the method disclosed in the embodiment of the present invention obtained better test results than the large text model obtained by the existing traditional pre-training method.
[0066] To further explain the principle, the large text model has different degrees of understanding of different data sets in different sub-fields of different professional fields (for example, for some difficult scenarios, the model is difficult to understand, etc.). In the traditional pre-training method of complete mixed random sampling, it is impossible to conduct centralized training for the above-mentioned difficult scenarios, and the model training process is not stable enough, resulting in poor test results of the trained model and the model is not robust enough. In view of the above situation, the multi-stage pre-training method proposed in the embodiment of the present invention grades the difficulty of the full amount of data (that is, the large amount of pre-training data sets originally obtained) (in the specific exemplary scheme, it can be divided into three levels: simple, medium, and difficult), and then performs multi-stage pre-training on the large text model based on the pre-training data sets of each preset difficulty level, and finally a large text model for a professional field with better test results and better robustness can be obtained.
[0067] In one embodiment of the present invention, the number of the preset rounds may be 1 round, and the purpose is only to allow the text model to have an understanding of the full amount of data.
[0068] In one embodiment of the present invention, in step 2, the step of classifying the difficulty of the full amount of data to obtain a pre-training data set of each preset difficulty level specifically includes:
[0069] Step 1), sampling the full amount of data in batches to obtain multiple batches of sampled data; wherein the step of sampling in batches may specifically include: for each data set in the full amount of data, sampling the data in each data set in sequence in a sequential sampling manner until the number of sampled data reaches a preset sampling threshold, to obtain a batch of sampled data; sampling in sequence until all the full amount of data sampling is completed, to obtain multiple batches of sampled data;
[0070] Step 2), input the sampled data of each batch into the trained text model, and calculate the cross entropy loss corresponding to the sampled data of each batch;
[0071] Step 3), based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, the sampled data of each batch is classified into difficulty, and a pre-trained data set of each preset difficulty level is obtained after difficulty classification.
[0072] See also Figure 2 and Figure 3 In a specific exemplary solution of an embodiment of the present invention, the steps of classifying the difficulty of the full amount of data and obtaining pre-training data sets of each preset difficulty level specifically include:
[0073] Input each batch of sampled data into the trained text model, calculate the cross entropy loss based on the actual output of the text model and the theoretical output of the data, and save it, and finally obtain the cross entropy loss of all batches of sampled data; in a specific exemplary scheme, all data sets in the general field and all data sets in different sub-fields of the power field can be sampled (exemplarily, K1 data in each batch) and the model loss is calculated to obtain the cross entropy loss on all batches of all data sets;
[0074] Based on the cross entropy loss corresponding to all batches of sampled data, the difficulty classification is performed using pre-set thresholds a and b; if the cross entropy loss is less than a, the sampled data of the corresponding batch is at a simple level; if the cross entropy loss is greater than or equal to a and less than b, the sampled data of the corresponding batch is at a medium level; if the cross entropy loss is greater than or equal to b, the sampled data of the corresponding batch is at a difficult level;
[0075] Based on the results of difficulty classification, the sampled data of each batch of the three difficulty levels of simple, medium and difficult are merged, and then the data of each difficulty level are mixed and randomly sampled when pre-training the large text model.
[0076] In the technical solution provided by the embodiment of the present invention, in order to better classify a large amount of general field pre-training data and professional field pre-training data according to the degree of model acceptance and reduce the manual intervention of experts, the present invention uses the cross-entropy loss of each training batch as the basis for division. Specifically explanatory, in the prior art, data difficulty is generally defined by experts (explanatory, specifically by experts checking the difficulty of each data set and manually labeling each data set as simple, medium, and difficult). The disadvantages of this method include: the difficulty classification of experts may not be applicable to the model, and some experts think that the difficult data may be easier to learn by the model; it requires experts in the corresponding field to invest a lot of energy, the manpower cost is high, and the expert classification has a certain subjectivity. Different from the prior art, the technical solution of the embodiment of the present invention classifies the difficulty level of each data set based on the cross-entropy loss of the large text model, which largely overcomes the defects of subjective classification by experts. It should also be emphasized that the technical solution of the embodiment of the present invention does not calculate the cross entropy loss for each data set as a whole, and then classify according to the loss; explanatory, this classification method that takes the entire data set as a whole does not classify each batch within the data set, and the classification result is relatively rough. The difficult data set classified by this classification method may also contain medium or simple data. The scheme of dividing the batch sampling data as a whole disclosed in the embodiment of the present invention can obtain more refined division results, which can better assist the convergence and fitting of the model.
[0077] In one embodiment of the present invention, based on the pre-training data sets of each preset difficulty level obtained in step 2, the text model after one training is pre-trained in stages from easy to difficult, and the steps of obtaining the pre-trained professional field text model specifically include:
[0078] Based on the pre-training data sets of each preset difficulty level obtained in step 2, the text model is trained from easy to difficult using the pre-training data sets of each preset difficulty level to obtain a trained text model;
[0079] Based on the pre-trained data sets of each preset difficulty level obtained in step 2, the above-trained text large model is trained again using the pre-trained data sets of each preset difficulty level and their combinations from easy to difficult using an incremental learning method to obtain a pre-trained professional field text large model.
[0080] See also Figure 4In a further exemplary scheme, the text model can be first trained using simple, medium, and difficult data in sequence, so that the model can "preview" the knowledge in the data set; then the incremental learning method is used to further train the model in three stages: simple, a combination of simple and medium, and a combination of simple, medium and difficult, so that when the model learns knowledge of a higher difficulty level, it can "review" the data of the previous difficulty level.
[0081] In the technical solution disclosed in the embodiment of the present invention, the learning process of human beings from elementary school, junior high school, high school, and university from easy to difficult is simulated, and the cross entropy loss of the text big model on each data set is used as the degree of understanding of the text big model on the data set, and the sampled data of each batch of each data set can be divided into multiple difficulty levels such as simple, medium, and difficult at a finer granularity. Compared with the existing traditional random sampling learning method of all data, the multi-stage pre-training method disclosed in the embodiment of the present invention performs sampling learning based on the mastery of the sampled data of each batch in each data set by the text big model itself, which can accelerate the convergence speed of model training. Compared with the traditional method in which the existing domain experts divide each data set into three difficulty levels of simple, medium, and difficult based on their own feelings about each data set, the technical solution disclosed in the embodiment of the present invention uses the difficulty cognition of the text big model itself as the classification basis, eliminating the gap between the feelings of human experts and the ability of the model itself; at the same time, the method of the embodiment of the present invention also reduces the intervention of human experts and reduces the investment in manpower costs.
[0082] To further explain specifically, compared with the solution in "Multi-stage Pre-training over SimplifiedMultimodal Pre-training Models", the technical solution disclosed in the embodiment of the present invention proposes a difficulty classification method for large text models, which uses a model loss function as the classification basis instead of using three classification levels of words, phrases, and sentences, and is more suitable for pre-training of large text models.
[0083] To further explain in detail, compared with the solution in "An Asynchronous Updating Reinforcement Learning Framework for Task-oriented Dialog System", in terms of difficulty classification method, the technical solution provided by the embodiment of the present invention makes up for the deficiency that the accuracy of the text cannot be calculated; in terms of training scheduling method, the technical solution disclosed in the embodiment of the present invention proposes a preview-incremental learning review training method, which can effectively prevent the model from forgetting knowledge and obtain a large text model with better testing effect.
[0084] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.
[0085] See also Figure 5 In an embodiment of the present invention, a multi-stage pre-training system for a large model of professional field text is provided, comprising:
[0086] A one-time training module is used to train the pre-trained text model for a preset number of rounds using a mixed random sampling method based on the acquired full amount of data to obtain a one-time trained text model;
[0087] A difficulty classification module is used to sample the full amount of data in batches, and calculate the cross entropy loss corresponding to each batch of sampled data based on the large text model after one training, and classify the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level;
[0088] The secondary training module is used to perform pre-training on the large text model after the first training in stages from easy to difficult based on the pre-training data sets of various preset difficulty levels, so as to obtain a large text model in a professional field after multi-stage pre-training.
[0089] In one embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the multi-stage pre-training method of the large model of professional field text.
[0090] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-stage pre-training method of the large model of professional field text in the above embodiment.
[0091] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0093] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A multi-stage pre-training method for a large text model in a professional field, characterized in that: The following steps are involved: Based on the acquired full amount of data, a pre-trained text model is trained for a preset number of rounds using a mixed random sampling method to obtain a trained text model. The full amount of data is sampled in batches, and the cross entropy loss corresponding to each batch of sampled data is calculated based on the large text model after one training, and the difficulty of the full amount of data is classified based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level; Based on pre-training data sets of various preset difficulty levels, the large text model after one training is pre-trained in stages from easy to difficult, and a large text model in a professional field is obtained after multi-stage pre-training.
2. According to claim 1, a multi-stage pre-training method for a large model of professional field text is characterized in that: The steps of sampling the full amount of data in batches, calculating the cross entropy loss corresponding to each batch of sampled data based on a large text model trained once, and classifying the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level include: The full amount of data is sampled in batches to obtain multiple batches of sampled data; wherein, in the process of sampling in batches, for each data set in the full amount of data, the data in the data set is sampled in sequence in a sequential sampling manner until the number of sampled data reaches a preset sampling threshold, thereby obtaining a batch of sampled data; Input each batch of sampled data into the trained text model, and calculate the cross entropy loss corresponding to each batch of sampled data; Based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, each batch of sampled data is classified by difficulty to obtain a pre-trained data set of each preset difficulty level after difficulty classification.
3. According to claim 2, a multi-stage pre-training method for a large professional domain text model is characterized in that: In the step of classifying the difficulty of each batch of sampled data based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, and obtaining a pre-trained data set of each preset difficulty level after difficulty classification, The preset classification thresholds are a and b, and a<b; When classifying the difficulty of each batch of sampled data, if the cross entropy loss is less than a, the corresponding batch of sampled data is at a simple level; if the cross entropy loss is greater than or equal to a and less than b, the corresponding batch of sampled data is at a medium level; if the cross entropy loss is greater than or equal to b, the corresponding batch of sampled data is at a difficult level; The batches of sampling data belonging to the simple level are merged, the batches of sampling data belonging to the medium level are merged, and the batches of sampling data belonging to the difficult level are merged to obtain pre-training data sets of each preset difficulty level after difficulty classification.
4. The multi-stage pre-training method for a large model of professional field text according to claim 1 is characterized in that: The steps of performing stage-by-stage pre-training from easy to difficult on the text model after one training based on the pre-training data sets of each preset difficulty level to obtain the professional field text model after multi-stage pre-training include: Using pre-trained data sets of various preset difficulty levels from easy to difficult, a mixed random sampling method is used to train the large text model after the first training to obtain the large text model after the second training; An incremental learning method is adopted to utilize pre-trained data sets of various preset difficulty levels and combinations of pre-trained data sets of various preset difficulty levels from easy to difficult, and a mixed random sampling method is adopted to train the large text model after secondary training to obtain a large text model in a professional field after multi-stage pre-training.
5. The multi-stage pre-training method for a large professional domain text model according to claim 1 is characterized in that: In the step of training the pre-trained text model for a preset number of rounds by adopting a mixed random sampling method based on the acquired full amount of data to obtain a trained text model, The number of the preset rounds is at least 1 round.
6. A multi-stage pre-training system for large text models in professional fields, characterized by: include: A one-time training module is used to train the pre-trained text model for a preset number of rounds using a mixed random sampling method based on the acquired full amount of data to obtain a one-time trained text model; A difficulty classification module is used to sample the full amount of data in batches, and calculate the cross entropy loss corresponding to each batch of sampled data based on the large text model after one training, and classify the difficulty of the full amount of data based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level; The secondary training module is used to perform pre-training on the large text model after the first training in stages from easy to difficult based on the pre-training data sets of various preset difficulty levels, so as to obtain a large text model in a professional field after multi-stage pre-training.
7. A multi-stage pre-training system for a large model of professional domain text according to claim 6, characterized in that: In the difficulty classification module, the steps of sampling the entire data in batches, calculating the cross entropy loss corresponding to each batch of sampled data based on the trained large text model, and classifying the entire data in difficulty based on the cross entropy loss corresponding to each batch of sampled data to obtain pre-trained data sets of each preset difficulty level include: The full amount of data is sampled in batches to obtain multiple batches of sampled data; wherein, in the process of sampling in batches, for each data set in the full amount of data, the data in the data set is sampled in sequence in a sequential sampling manner until the number of sampled data reaches a preset sampling threshold, thereby obtaining a batch of sampled data; Input each batch of sampled data into the trained text model, and calculate the cross entropy loss corresponding to each batch of sampled data; Based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, each batch of sampled data is classified by difficulty to obtain a pre-trained data set of each preset difficulty level after difficulty classification.
8. A multi-stage pre-training system for a large model of professional domain text according to claim 7, characterized in that: In the step of classifying the difficulty of each batch of sampled data based on the cross entropy loss corresponding to each batch of sampled data and the preset classification threshold, and obtaining a pre-trained data set of each preset difficulty level after difficulty classification, The preset classification thresholds are a and b, and a<b; When classifying the difficulty of each batch of sampled data, if the cross entropy loss is less than a, the corresponding batch of sampled data is at a simple level; if the cross entropy loss is greater than or equal to a and less than b, the corresponding batch of sampled data is at a medium level; if the cross entropy loss is greater than or equal to b, the corresponding batch of sampled data is at a difficult level; The batches of sampling data belonging to the simple level are merged, the batches of sampling data belonging to the medium level are merged, and the batches of sampling data belonging to the difficult level are merged to obtain pre-training data sets of each preset difficulty level after difficulty classification.
9. The multi-stage pre-training system for a large model of professional domain text according to claim 6, characterized in that: In the secondary training module, the steps of executing pre-training data sets based on various preset difficulty levels, performing stage-by-stage pre-training from easy to difficult on the large text model after the primary training, and obtaining the large professional field text model after multi-stage pre-training include: Using pre-trained data sets of various preset difficulty levels from easy to difficult, a mixed random sampling method is used to train the large text model after the first training to obtain the large text model after the second training; An incremental learning method is adopted to utilize pre-trained data sets of various preset difficulty levels and combinations of pre-trained data sets of various preset difficulty levels from easy to difficult, and a mixed random sampling method is adopted to train the large text model after secondary training to obtain a large text model in a professional field after multi-stage pre-training.
10. The multi-stage pre-training system for a large model of professional domain text according to claim 6, characterized in that: In the one-time training module, a step is performed in which a pre-trained text model is trained for a preset number of rounds by adopting a mixed random sampling method based on the acquired full amount of data to obtain a trained text model. The number of the preset rounds is at least 1 round.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the multi-stage pre-training method for a large professional field text model as described in any one of claims 1 to 5.
12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the multi-stage pre-training method for a large professional domain text model as described in any one of claims 1 to 5.
Citation Information
Cited By
Target detection method and device based on domain self-adaption, equipment and medium
CN121708557A
