Language model training method and device, storage medium and computer program product

Abstract

The invention discloses a language model training method and device, a storage medium and a computer program product, and relates to the technical field of natural language processing, and the method comprises the steps: after a second device trains a small language model, sending a second training result of the small language model for text data to a first device; the first device determines that the text data belongs to a second device whose corresponding second training result satisfies a preset condition, and sends first sending indication information to the second device, and the second device sends a second probability distribution prediction result of the text data indicated in the first sending indication information to the first device, and the first device optimizes the large language model based on the second probability distribution prediction result until a preset training ending condition is met, and then a trained large language model is obtained. The communication traffic transmitted between the first device and the second device can be optimized in the model training process.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a language model training method, device, storage medium and computer program product. Background Art

[0002] With the rapid development of large language model (LLM) technology, their powerful performance has brought significant productivity improvements in all walks of life. In the process of training and using large language models, a large amount of computing resources are required, which means that only a few large enterprises are able to train and fully utilize these models. Due to the limitation of computing resources, small and medium-sized enterprises cannot directly run large language models with particularly large model parameter scales, and can only run large language models with relatively limited model parameter scales (for distinction, they can also be called small language models). At the same time, as the model parameter scale of large language models becomes larger and larger, more and more training text data will be required. At present, open source training text datasets have gradually failed to meet the training and fine-tuning of large language models, especially some text data in specific fields are in various enterprises. At this time, the collaborative fine-tuning of federated large and small models is a realistic and meaningful technical means.

[0003] In the existing federated large and small model collaborative fine-tuning algorithm, an enterprise device that deploys a large language model (for illustration, referred to as the first device) needs to receive the text probability distribution prediction results (i.e., the predicted probability corresponding to each text component unit in the predicted output text) of each public text data sent by each enterprise device that deploys a small language model (for illustration, referred to as the second device). During the entire training process, the communication volume between the first device and the second device is relatively large, and the network bandwidth requirement is relatively high.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a language model training method, device, storage medium and computer program product to optimize the communication volume transmitted between the first device and the second device in the fine-tuning of the federated size model.

[0006] To achieve the above object, the present application proposes a language model training method, which is applied to a first device, wherein the first device is respectively connected to each second device for communication, a large language model to be trained is deployed in the first device, and a small language model to be trained is respectively deployed in each second device, and the method includes:

[0007] receiving a second training result respectively sent by each of the second devices, where the second training result is a training result generated by the second device based on a second output text prediction result, where the second output text prediction result is a result obtained by the second device training the small language model using a local private text sample, and then inputting input text data in a public text sample into the trained small language model for prediction;

[0008] Determine a target second training result and a corresponding target second device that meet a preset condition, the public text sample belongs to the target second device, and the number of the target second devices is less than the number of second devices participating in the training;

[0009] Sending first sending indication information to the target second device, wherein the first sending indication information is used to instruct the target second device to send a second text probability distribution prediction result of the input text data in the public text sample to the first device, wherein the second text probability distribution prediction result is a result obtained by the target second device inputting the input text data into the trained small language model for prediction;

[0010] Receiving the second text probability distribution prediction result sent by the target second device, and training the large language model based on the second text probability distribution prediction result;

[0011] Return to the above step of receiving the second training results sent by each of the second devices respectively, until the preset training end condition is met, and obtain a large language model that has completed the training.

[0012] Optionally, before the step of determining the target second training result and the corresponding target second device that meet the preset condition, the step further includes:

[0013] Inputting input text data in the public text sample held by the first device into the large language model for prediction to obtain a first output text prediction result;

[0014] Generate a first training result based on the first output text prediction result;

[0015] The step of determining a target second training result and a corresponding target second device that meet a preset condition comprises:

[0016] Performing a comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result;

[0017] Selecting a target second training result that meets a preset condition based on the first comparison result, wherein the target second training result is better than the first training result in quality;

[0018] Determine that the second device corresponding to the target second training result is the target second device.

[0019] Optionally, the public text sample includes multiple items;

[0020] The step of selecting a target second training result that meets a preset condition based on the first comparison result includes:

[0021] For each of the public text samples, based on the first comparison result corresponding to the public text sample, determining the optimal training result corresponding to the input text data in the public text sample, the optimal training result being the training result with the best quality among the first training result corresponding to the input text data and all the corresponding second training results;

[0022] The second training result that meets a preset condition is used as the target second training result, and the preset condition is that the second training result is the optimal training result corresponding to the input text data.

[0023] Optionally, the second training result includes the second output text prediction result; and the step of generating the first training result based on the first output text prediction result includes:

[0024] Generate a first training result including the first output text prediction result;

[0025] The step of performing comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result includes:

[0026] The input text data, the first output text prediction result in the first training result, and the second output text prediction results in each of the second training results are input into a preset question-answering model, so that the question-answering model evaluates the matching degree between the first output text prediction result, each of the second output text prediction results and the input text data, and outputs the first comparison result based on the matching degree.

[0027] Optionally, the second training result includes a second prediction effect metric index value, where the second prediction effect metric index value is a metric index value of a prediction effect of the small language model of the second device on the input text data, calculated by the second device based on the second output text prediction result and the label text data in the public text sample;

[0028] The step of generating a first training result based on the first output text prediction result comprises:

[0029] Based on the first output text prediction result and the label text data in the public text sample, calculating a metric value of the prediction effect of the large language model on the input text data to obtain a first prediction effect metric value;

[0030] Generate a first training result including the first prediction effect measurement indicator value;

[0031] The step of performing comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result includes:

[0032] The first prediction effect measurement index value in the first training result and the second prediction effect measurement index value in each of the second training results are compared according to the size of the index value to obtain the first comparison result.

[0033] In addition, to achieve the above purpose, the present application also proposes a language model training, which is applied to any second device among multiple second devices, each of which is respectively connected to the first device for communication, a large language model to be trained is deployed in the first device, and a small language model to be trained is deployed in each of the second devices, and the method includes:

[0034] Using a local private text sample to train the small language model, and based on the trained small language model, predicting the input text data in the public text sample to obtain a second output text prediction result and a second text probability distribution prediction result;

[0035] Generate a second training result based on the second output text prediction result;

[0036] The second training result is sent to the first device, so that after the first device determines the target second training result and the corresponding target second device that meet the preset conditions, the first device sends a first sending indication information to each second device, the public text sample belongs to the target second device, the number of the target second devices is less than the number of second devices participating in the training, and the first sending indication information is used to instruct the target second device to send the second text probability distribution prediction result of the input text data in the public text sample to the first device, and the second text probability distribution prediction result is the result obtained by the target second device inputting the input text data into the trained small language model for prediction;

[0037] Upon receiving the first sending indication information sent by the first device, sending a second text probability distribution prediction result of the input text data to the first device, so that the first device trains the large language model based on the second text probability distribution prediction result;

[0038] receiving a third training result sent by the first device, wherein the third training result is a training result generated based on the third output text prediction result after the first device inputs the input text data in the public text sample into the trained large language model for prediction to obtain a third output text prediction result;

[0039] Performing a comparative analysis on the training quality of the second training result and the third training result, and in the case where it is determined that the quality of the third training result is better than the quality of the second training result, sending second sending indication information to the first device, wherein the second sending indication information is used to instruct the first device to send a first text probability distribution prediction result of the input text data to the second device, wherein the first text probability distribution prediction result is a result obtained by the first device inputting the input text data into the trained large language model for prediction;

[0040] Receiving the first text probability distribution prediction result sent by the first device, and optimizing the small language model based on the first text probability distribution prediction result;

[0041] Return to the above step of training the small language model using local private text samples until the preset training end condition is met, thereby obtaining a trained small language model.

[0042] Optionally, after the step of performing comparative analysis on the training quality of the second training result and the third training result, the step further includes:

[0043] In a case where it is determined that the quality of the second training result is better than the quality of the third training result, the small language model is optimized based on the second text probability distribution prediction result.

[0044] Optionally, the third training result includes the third output text prediction result; and the step of generating the second training result based on the second output text prediction result includes:

[0045] Generate a second training result including the second output text prediction result;

[0046] The step of performing comparative analysis on the training quality of the second training result and the third training result comprises:

[0047] The input text data, the second output text prediction result in the second training result, and the third output text prediction result in the third training result are input into a preset question-answering model, so that the question-answering model evaluates and outputs a training result with the best training quality for the input text data.

[0048] Optionally, the third training result includes a third prediction effect metric index value, where the third prediction effect metric index value is a metric index value of a prediction effect of the large language model of the first device on the input text data, calculated by the first device based on the third output text prediction result and the label text data in the public text sample;

[0049] The step of generating a second training result based on the second output text prediction result comprises:

[0050] Based on the second output text prediction result and the label text data in the public text sample, calculating a metric value of the prediction effect of the small language model of the second device on the input text data to obtain a second prediction effect metric value;

[0051] Generate a second training result including the second prediction effect metric value;

[0052] The step of performing comparative analysis on the training quality of the second training result and the third training result comprises:

[0053] The second prediction effect measurement index value in the second training result and the third prediction effect measurement index value in the third training result are compared according to the size of the index value.

[0054] In addition, to achieve the above-mentioned purpose, the present application also proposes a language model training device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the language model training method described above.

[0055] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the language model training method described above are implemented.

[0056] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the language model training method described above.

[0057] One or more technical solutions proposed in this application have at least the following technical effects:

[0058] By deploying a large language model to be trained on a first device and a small language model to be trained on a second device, the first device uses input text data to coordinate with each second device to use input text data and their own private text data to collaboratively train the large language model and the small language model, thereby realizing a collaborative training architecture of large language models and small language models, so that enterprises with limited computing resources can also participate in the training of large language models, and use their private text data in subdivided fields to enhance the natural language processing capabilities of the large language model; wherein, in each round of training, after each second device uses its own private text sample to locally train the small language model, the input text data in the public text sample is input into the small language model for prediction to obtain a second output text prediction result, and a second training result is generated based on the second output text prediction result, and the second training result is sent to the first device, so that without exposing the privacy of the private text data of each second device, the private text data in the second device is used to help optimize the large language model in the first device; and the small language model is deployed in each second device. , so the computing power requirement for the second device is also relatively low, so that most companies with limited computing power resources can also participate in the training of the language model to obtain a language model with stronger natural language processing capabilities; then the first device determines the second training result that meets the preset conditions and the corresponding target second device, and determines that the public text sample belongs to the target second device, the number of target second devices is less than the number of second devices participating in the training, and sends a first sending indication information to the target second device, the first sending indication information is used to instruct the target second device to send the second text probability distribution prediction result of the input text data in the public text sample to the first device, and analyzes the training results of the public text sample by the small language model of each second device, so as to select some second devices from all second devices to send the text probability distribution information for the public text sample learned by their respective small language models, without the need for all second devices to transmit the text probability distribution information of each public text sample learned by their respective small language models, which can reduce the communication volume sent by the second device and the communication volume received by the first device, thereby reducing the load pressure of the second device. Afterwards, by receiving the second text probability distribution prediction result sent by the target second device, and training the large language model based on the second text probability distribution prediction result, since the training result of the small language model of the target second device selected for each public text sample meets the preset conditions for the public text sample, the knowledge learned by the small language model with better natural understanding ability of the input text data can be selected for learning, thereby improving the natural language processing ability of the large language model.Furthermore, in the next round of training, based on the enhanced small language model, it is possible to further learn the privacy knowledge in the field of each enterprise and then feed it back to the large language model. This iterative cycle allows a large language model with stronger natural language processing capabilities to be obtained after multiple rounds of training. In addition, this embodiment makes full use of the computing resources of each device and the text data of the private subdivision field of the corresponding enterprise to realize the training of the model language, and realizes that when the computing power resources of the enterprise are limited, the large language model absorbs the privacy knowledge in the field of each enterprise, while also protecting the data privacy of each enterprise. Therefore, in this embodiment, by selecting only the second device corresponding to the small language model with a better learning effect on the public text sample for each public text sample to transmit the text probability distribution information, the communication volume received by the second device during the model training process can be optimized, and at the same time, the large language model can learn the knowledge of each private field, thereby improving the natural language processing ability of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0061] Figure 1 A flowchart of the first embodiment of the language model training method of the present application is provided;

[0062] Figure 2 A flowchart of the third embodiment of the language model training method of the present application is provided;

[0063] Figure 3 A schematic diagram of one of the application scenarios involved in the language model training process of this application;

[0064] Figure 4 A schematic diagram of another application scenario involved in the language model training process of this application;

[0065] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the language model training method in the embodiment of the present application.

[0066] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0068] The following is a first embodiment of the language model training method of the present application. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the language model training method of this application. In this embodiment, the language model training method is applied to the first device, and the first device is connected to each second device in communication. Both the first device and the second device can be a computing service device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, a mobile phone, etc. The first device and the second device can be devices deployed in different enterprises, that is, each enterprise uses its own device to jointly train the language model to perform natural language processing tasks through the trained language model. In a specific implementation, specific natural language processing tasks can be set according to the specific needs of each enterprise. For example, in a dialogue scenario, a task of generating a reply for the above text is set, and in a summary generation scenario, a task of generating a text summary for the text is set. The number of second devices is at least two, and the operations performed or functions implemented by the first device and the second device in the process of training the language model are different, and the operations performed or functions implemented by each second device are the same. Therefore, in the following embodiments, the two devices are distinguished as "first" and "second". In some application scenarios, the first device can also be called a server or a coordination end, etc., and the second device can be called a client or a participating end, etc.

[0069] A large language model to be trained is deployed in the first device, and a small language model is deployed in each second device. A language model is a model that can be used to perform natural language processing tasks, such as LLaMa2 13B, LLaMa2 7B, OPT 7B, etc. In this embodiment, both the large language model and the small language model belong to language models. The large language model has more parameters, and the small language model has fewer parameters. The large language model and the small language model can be isomorphic, that is, they have the same model architecture, but different scales of model parameters. For example, in one feasible implementation, LLaMa2 13B and LLaMa2 7B can be deployed in the first device, and LLaMa2 7B can be deployed in the second device. Due to the difference in the number of parameters, the large language model after training is generally better than the small language model after training in terms of its ability to process natural language tasks, but at the same time, the computing resources required for the training process and the operation process after training of the large language model are also more than those of the small language model. In the specific implementation process, an enterprise with strong computing power can provide equipment as the first equipment, and an enterprise with text data in various segments but limited computing power can provide equipment as the second equipment. The enterprises jointly train the language model and use the advantages of both parties to jointly train to obtain a language model with higher natural language processing capabilities.

[0070] The model to be trained refers to the model parameters in the model that need to be optimized through the training process, and its initial model parameters can be obtained after model pre-training (or the pre-trained model can be used as the model to be trained), or, its initial model parameters can also be initialized according to experience or randomly initialized. The process of using a pre-trained model as the model to be trained and training the model to be trained on a specific natural language processing task is also called "fine-tuning", and "training" is used to represent this process in the following embodiments. In a specific implementation, a pre-trained language model is used as the language model to be trained, and the language model to be trained is further trained, so that a better training effect can be obtained, that is, the trained language model can have a stronger natural language processing capability. In this embodiment, it is not limited to using a pre-trained language model as the language model to be trained, nor is it limited to a specific pre-training method.

[0071] The first device and the second device are respectively deployed with text samples for training the language model. Generally speaking, there are multiple text samples for training the language model, which constitute a training data set. Each text sample includes input text data as model input and label text data as model training labels. For example, in a dialogue scenario, it is necessary to train the language model to generate corresponding answers based on specific questions. Then, the input text data can be questions and the label text data can be answers. The first device and the second device can deploy the text data for training the language model that has been made public in the industry (hereinafter referred to as "public text samples") for joint training of large language models and small language models. Each second device deploys the private text data of each enterprise (hereinafter referred to as private text samples) for local training of the small language model in the second device. The private text samples of the enterprise can be business data from the enterprise. Since the specific business directions of each enterprise are different, the private text data of each enterprise belongs to different subdivisions. Each second device participates in the training of the language model through the private text data of its own subdivision, which can provide the language model with knowledge of different subdivisions, so that the language model finally trained has stronger natural language processing capabilities. For example, the first device can be a device with computing power that can support the training and operation of a large language model, provided by a small number of enterprises or institutions that can support the device; the input text data can be text data publicly available in the industry for training language models; each second device can be deployed in banks, insurance companies and other enterprises, and private text samples can be business data from each enterprise, such as conversation business data from customer service personnel of banks and insurance companies when serving customers. These private text samples belong to text data in subdivided fields such as banking business fields and insurance business fields. It should be noted that in the specific implementation method, the business data used needs to be agreed by the customer and can be desensitized to avoid leaking the customer's privacy.

[0072] In this embodiment, the language model training method includes steps S10 to S50:

[0073] Step S10, receiving the second training results sent by each second device respectively, the second training result is a training result generated by the second device based on the second output text prediction result, and the second output text prediction result is a result obtained by the second device using a local private text sample to train a small language model, and then inputting the input text data in the public text sample into the trained small language model for prediction.

[0074] The first device and each second device jointly perform multiple rounds of iterative training on the large language model to be trained and the small language model to be trained based on their respective computing resources, and each round of training is performed on the basis of the previous round of training. In a round of training, each second device uses its own local private text samples to train its own small language model based on its computing resources, trains (or optimizes) the small language model, and then inputs the input text data in the public text sample into the trained small language model. The trained small language model predicts the input text data and outputs a prediction result (hereinafter referred to as the "second prediction result" for distinction). The second prediction result may include a second output text prediction result and a second text probability distribution prediction result, and a second training result is generated based on the second output text prediction result. The second output text prediction result is the result obtained by the small language model predicting the output text corresponding to the input text data. For example, the input text data is "where does the sun rise from", and the second output text prediction result can be the answer sentence "the sun rises from the east" for the input text data. The second text probability distribution prediction result is the output text probability distribution predicted by the small language model, that is, the predicted probability of each text component unit of the output text in the vocabulary corresponding to the small language model (hereinafter the vocabulary corresponding to the small language model is referred to as the "second vocabulary").

[0075] The first device can receive the second training results for the input text data in the public text sample respectively sent by each second device. In a specific implementation, the second training result may include at least one of the second output text prediction result and the second prediction effect metric value, and the second prediction effect metric value is the metric value of the prediction effect of the small language model of the second device on the input text data calculated by the second device based on the second output text prediction result and the label text data in the public text sample, or the second training result may include the second output text prediction result and the second prediction loss. Specifically, which training results are included can be set as needed, and there is no limitation on this in the present embodiment. In addition, the second prediction effect metric value may be a cross entropy, a sample prediction accuracy rate, or other prediction effect metric, and there is no limitation on this in the present embodiment.

[0076] It should be noted that the local private text samples of each second device come from the subdivision fields to which each enterprise belongs. After each second device uses its own local private text samples to train its own small language model, each small language model learns the text information in the corresponding subdivision field (which can be understood as a kind of "knowledge" learned by the model). The second prediction result obtained by each second device using its own locally trained small language model to process the same input text data is the prediction result made by each small language model based on its own learned "knowledge".

[0077] The first device receives the second training results sent by each second device, and performs the next training process based on the second training results. Since the second training results are only the prediction results made by the small language model in the second device for the input text data, the second device does not need to send its private text samples during the collaborative training with the first device, so the private text data will not be directly exposed to the first device, thereby using the private text samples in the second device to help optimize the large language model in the first device, so that the large language model can learn the text information of each segmented field while ensuring the privacy security of the private text data in the second device. In addition, since the structure of the small language model is relatively simple and the number of parameters is relatively small, the computing power requirements for the second device are also low, so that most companies with limited computing power resources can also participate in the training of the language model based on the text data in their respective segments to obtain a language model with stronger natural language processing capabilities.

[0078] For example, each second device deployed in a bank, insurance company and other enterprise uses private text data from the conversation business data of the bank, insurance company and other enterprises to perform local training on their respective small language models, so that each small language model learns the text information of the banking business field, the insurance business field and other sub-sectors, and sends the second training result obtained by training to the first device for the training of the large language model in the first device. The private text data of each enterprise itself will not be exposed, thereby protecting the privacy and security of the private text data of each enterprise. At the same time, the computing power requirements for the second device of each enterprise are also relatively low, so that the text data of the banking business field, the insurance business field and other sub-sectors can participate in the training of the large language model, and a language model with stronger natural language processing capabilities can be obtained.

[0079] Step S20, determining the second training results that meet the preset conditions and the corresponding target second devices, the public text samples belong to the target second devices, and the number of target second devices is less than the number of second devices participating in the training.

[0080] After receiving the second training results sent by each second device, the first device can compare and analyze the second training results of the small language models of each second device for the input text data in terms of training quality, so as to determine the device to which the public text sample belongs based on the comparison and analysis results. In some embodiments, the first device can screen out one or more target second devices whose corresponding second training results meet the preset conditions from all the second devices participating in the training, and determine that the public text sample belongs to one or more target second devices, thereby reducing the amount of communication received by the second device. In this case, the preset condition can be: the second training result ranks in the top n in quality, and n is less than the device of the second device participating in the training. The quality of the second training result can be analyzed in combination with a specific scenario. For example, when the second training result is a loss value obtained based on the second output text prediction result and the input text data in the public text sample, the quality of each second training result is compared and analyzed based on the size of the loss value. In other embodiments, the first device can predict the input text data in the public text sample in combination with the locally deployed large language model to obtain a first output text prediction result, and generate a first training result based on the first output text prediction result, and then select a training result with the best quality from the first training result and all the second training results received, and determine that the device corresponding to the training result with the best quality is the only device to which the public text sample belongs. In this case, the preset condition can be: the training result with the best quality among all training results (including the first training result and all the second training results received by the first device). When the training result with the best quality is any second training result, the first device can determine that the second training result meets the preset condition, and the second device corresponding to the second training result is the target second device, and the first device sends a first sending indication information to the target second device to instruct the target second device to send the text probability distribution prediction result of the public text sample to the first device; when the training result with the best quality is not any second training result, the first device can send a first static indication information to the second device to indicate that the second device does not need to send the text probability distribution prediction result of the public text sample.

[0081] It should be noted that in the existing fine-tuning training process of the federated size model, in each round of training, the first device needs to receive the text probability distribution information of all public text data sent from all second devices. At this time, the amount of communication received by the first device = the number of second devices * the number of training iterations * the number of public text samples * the length of the text after word segmentation corresponding to the input text data * the second vocabulary floating point number. The amount of communication received by the first device is relatively large, and the network bandwidth requirement is relatively high. Through this embodiment, the first device selects some second devices to transmit the text probability distribution information of the input text data learned by its small language model based on the training results of the small language model of each second device for the public text data, which significantly reduces the total amount of data transmission. At the same time, the amount of data received by the first device in a short period of time is greatly reduced, which can reduce the load pressure of the first device.

[0082] It should also be noted that different language models may have different natural understanding capabilities or learning effects for the same input text data. In this embodiment, by selecting the target second device whose corresponding second training result meets the preset conditions, the knowledge learned by the small language model with better natural understanding ability for the input text data can be selected for learning, which can significantly improve the training effect of the large language model and make it show higher accuracy in natural language processing.

[0083] In one embodiment, the second training result may be a second output text prediction result, then in the above step S20, determining the target second training result and the corresponding target second device that meet the preset conditions may include: the first device uses a preset question-answering model to perform a comparative analysis of the training quality of the second output text prediction results corresponding to each second device, so as to determine a second output text prediction result with the best quality among multiple second output text prediction results, and determine that the second output text prediction result with the best quality meets the preset conditions, the second training result corresponding to the second output text prediction result is the target second training result, and the second device corresponding to the target second training result is the target second device. Specifically, the question-answering model may be a third-party question-answering model, which is used to evaluate the matching degree of each second output text prediction result with the input text data, and determine the second output text prediction result that meets the preset conditions based on the matching degree.

[0084] In another embodiment, the second training result may include a second prediction effect measurement index value, which is an index value calculated by the second device based on the second output text prediction result and the label text data in the public text sample. In the above step S20, determining the target second training result and the corresponding target second device that meet the preset conditions may include: the first device sorts the second prediction effect measurement index values ​​in each second training result based on the size of the index value, and determines that the top n second effect measurement index values ​​meet the preset conditions, the second training result corresponding to the top n second effect measurement index values ​​is the target second training result, and the second device corresponding to the target second training result is the target second device.

[0085] Step S30, sending a first sending indication message to the target second device, the first sending indication message is used to instruct the target second device to send a second text probability distribution prediction result of the input text data in the public text sample to the first device, the second text probability distribution prediction result is the result obtained by the target second device inputting the input text data into the trained small language model for prediction.

[0086] When the first device determines the target second device to which the input text data in the public text sample belongs, the first device generates a first sending indication message based on the sample identifier or sample index of the public text sample assigned to each target second device. The first sending indication message is used to instruct the target second device to send a text probability distribution prediction result (hereinafter referred to as a "second text probability distribution prediction result" for distinction) obtained by a small language model of the target second device for predicting the assigned input text data to the first device. In some embodiments, the first device may send first sending indication information via a first notification message. The first notification message may be in the form of a vector, and the size of the vector is N*1, where N is the number of public text samples. The i-th variable in the vector represents the first indication information of the i-th public text sample. When the value of the i-th variable is 1, it represents the first sending indication information for the i-th public text sample, used to indicate that the second device needs to send a second text probability distribution prediction result for the input text data in the public text sample obtained by the small language model of the second device to the first device; when the value of the i-th variable is 0, it represents the first static indication information for the i-th public text sample, used to indicate that the second device does not need to send a second text probability distribution prediction result for the input text data in the public text sample obtained by the small language model of the second device to the first device.

[0087] Step S40: receiving a second text probability distribution prediction result sent by the target second device, and training a large language model based on the second text probability distribution prediction result.

[0088] If the preset training end condition is met, the first device executes step S50: obtaining a trained large language model.

[0089] If the preset training end condition is not met, the process returns to step S10.

[0090] When the first device receives the second text probability distribution prediction result for the input text data in the public text sample sent by the target second device, it trains the large language model based on the second text probability distribution prediction result of the input text data, that is, updates the parameters of the large language model. Specifically, the second text probability distribution prediction result predicted for the input text data in the public text sample learned by the small language model of the target second device (which can be understood as the knowledge of the input text data learned by the small language model of the target second device) is used to perform knowledge distillation from the small language model of the target second device to the large language model to optimize the large language model. Knowledge distillation (KD) refers to a model training method that transfers the knowledge of the teacher model to the student model. In step S40 of the scheme of this embodiment, the first device uses the small language model of the target second device as the teacher model, the large language model as the student model, and the second text probability distribution prediction result of the input text data learned by the small language model of the target second device as the soft target, so that the large language model fits the small model of the target second device in the direction of the soft target, so that the large model learns a function mapping similar to the small model, thereby achieving the purpose of simulating the performance of the large model.

[0091] Specifically, when there is only one target second device assigned to the public text sample, the first device can train the large language model based on the second text probability distribution prediction result uniquely corresponding to the public text sample. When there are multiple target second devices assigned to the public text sample (multiple means greater than or equal to two), the first device can randomly select a second text probability distribution prediction result from the second text probability distribution prediction results corresponding to multiple second devices to train the large language model, or the first device can perform weighted averaging on the second text probability distribution prediction results corresponding to multiple second devices to obtain an average text probability distribution prediction result, and train the large language model based on the average text probability distribution prediction result. In other cases, if the device to which the public text sample belongs is not any second device, the first device trains the large language model based on the text probability distribution prediction result predicted by the local large language model for the public text sample.

[0092] It should be noted that in each round of training, the first device selects the knowledge information of the input text data learned by the small language model of some second devices from all the second devices participating in the training based on the second training results corresponding to each second device for learning. Therefore, it is not necessary for each second device to send the text probability distribution information of the input text data learned by its own small language model to the first device, which can reduce the amount of communication received by the first device, and because the amount of communication data received by the first device in a short time is greatly reduced, the load pressure of the first device can be reduced. At the same time, since the second training result of the target second device selected by the first device for the input text data meets the preset conditions, the small language model learned by the large language model is more accurate for the knowledge of a certain subdivision field, thereby improving the effect of the large language model training. In addition, in the present embodiment, the process of knowledge distillation from the small language model to the large language model is carried out in the first device, and the participation of each second device is not required. Therefore, it can achieve the purpose of avoiding the disadvantage of limited hardware conditions of most enterprises and using the text data owned by each enterprise with limited hardware conditions to jointly train the large language model to obtain a language model with stronger natural language processing capabilities.

[0093] After the first device trains the large language model based on the prediction result of the second text probability distribution, it can detect whether the preset training end condition is met. The preset training end condition is a condition that needs to be met to end the pre-training training, which can be set as needed, and is not limited in this embodiment. For example, in a specific implementation, it can be set to the convergence of the loss function of the large language model or the number of training rounds reaching the set number of rounds, or the training duration reaching the set duration, etc.

[0094] If the first device detects that a preset training end condition is met, the iterative training process can be ended, and the final updated large language model is used as the large language model after training.

[0095] In this embodiment, a large language model to be trained is deployed on the first device, and a small language model to be trained is deployed on the second device. The first device uses input text data to cooperate with each second device to use input text data and their own private text data to collaboratively train the large language model and the small language model, thereby realizing a collaborative training architecture of the large language model and the small language model, so that enterprises with limited computing resources can also participate in the training of the large language model, and use their private text data in the subdivided fields to enhance the natural language processing capability of the large language model. For example, enterprises such as banks and insurance companies can use their devices with limited computing power to participate in the collaborative training of the large language model and the small language model, and use text data in the subdivided fields such as the banking business field and the insurance business field to enhance the natural language processing capability of the large language model; wherein, in each round of training, after each second device uses its own private text sample to locally train the small language model, the input text data in the public text sample is input into the small language model for prediction to obtain a second output text prediction result, and a second training result is generated based on the second output text prediction result, and the second training result is sent to the first device, so that the use of the private text data of each second device is not exposed to the privacy of the private text data. The private text data in the second device helps optimize the large language model in the first device; and, a small language model is deployed in each second device, so the computing power requirement for the second device is also relatively low, so that most enterprises with limited computing power resources can also participate in the training of the language model to obtain a language model with stronger natural language processing capabilities; then the first device determines the second training result and the corresponding target second device that meet the preset conditions, and determines that the public text sample belongs to the target second device, the number of target second devices is less than the number of second devices participating in the training, and sends a first sending indication information to the target second device, the first sending indication information is used to instruct the target second device to send the second text probability distribution prediction result of the input text data in the public text sample to the first device, and analyzes the training results of the public text sample by the small language model of each second device, so as to select some second devices from all second devices to send the text probability distribution information for the public text sample learned by their respective small language models, without the need for all second devices to transmit the text probability distribution information of each public text sample learned by their respective small language models, which can reduce the communication volume sent by the second device and the communication volume received by the first device, thereby reducing the load pressure of the second device.Afterwards, by receiving the second text probability distribution prediction result sent by the target second device, and training the large language model based on the second text probability distribution prediction result, since the training result of the small language model of the target second device selected for each public text sample for the public text sample meets the preset conditions, it is possible to select the knowledge learned by the small language model with better natural understanding ability of the input text data for learning, thereby improving the natural language processing ability of the large language model. Further, in the next round of training, on the basis of the enhanced small language model, it is possible to further learn the privacy knowledge in the field of each enterprise and then feed it back to the large language model, so that after multiple rounds of training, a large language model with stronger natural language processing ability is finally obtained. In addition, this embodiment makes full use of the computing resources of each device and the text data of the private subdivision field of the corresponding enterprise to realize the training of the model language, and realizes that when the computing power resources of the enterprise are limited, the large language model absorbs the privacy knowledge in the field of each enterprise, and also protects the data privacy of each enterprise. Therefore, in this embodiment, by only selecting the second device corresponding to the small language model that has a better learning effect on each public text sample to transmit the text probability distribution information, the communication volume received by the second device during the model training process can be optimized, and at the same time, the large language model can learn various private domain knowledge, thereby improving the natural language processing capabilities of the large language model.

[0096] Based on the above-mentioned first embodiment, a second embodiment of the language model training method of the present application is proposed. In this embodiment, the same or similar contents as those of the above-mentioned first embodiment can be referred to the above introduction and will not be repeated later. In this embodiment, for any public text sample, on the basis of the second training results for the public text sample learned by the small language models of each second device, combined with the training results of the large language model of the first device on the input text data in the public text sample, the target second device to which the public text sample belongs is selected to improve the accuracy of the knowledge learned by the large language model, thereby further improving the natural language processing capabilities of the trained large language model. Specifically, the above-mentioned step S20 may also include steps S12 to S13:

[0097] Step S12: inputting the input text data in the public text sample held by the first device into the large language model for prediction to obtain a first output text prediction result.

[0098] The first device may input the input text data in the locally held public text sample into the large language model for prediction, so as to evaluate the natural language processing capability of the large language model for the input text data in the public text sample based on the predicted first output text prediction result.

[0099] Step S13: generating a first training result based on the first output text prediction result.

[0100] After obtaining the first output text prediction result, the first device can use the first output text prediction result as the first training result, or can calculate a metric value of the prediction effect of the large language model on the input text data in the public text sample based on the first output text prediction result and the label text data in the public text sample, and then use the calculated prediction effect metric value as the first training result, or set the first training result to include the calculated prediction effect metric value and the first output text prediction result.

[0101] Accordingly, the above step S20 may include steps S201 to S202:

[0102] Step S201, performing a comparative analysis on the training quality of the first training result and the second training results corresponding to each second device to obtain a first comparative result.

[0103] The first device can adopt a corresponding quality assessment strategy for the specific form of the first training result to perform a comparative analysis on the training quality of the first training result corresponding to the input text data and the corresponding second training results to obtain a first comparison result. The first comparison result may include a quality ranking result of the first training result and the second training results, so as to determine the difference in learning effects of different language models on the public text sample based on the first comparison result.

[0104] Step S202: selecting a target second training result that meets a preset condition based on the first comparison result, where the target second training result is better in quality than the prediction quality of the first training result.

[0105] After the first device obtains the first comparison result, the first comparison result may include quality comparison results of each second training result and the first training result, and the first device may screen out one or more target second training results having better quality than the first training result.

[0106] In the above scheme, the second training results for the public text samples generated by the small language models of each second device and the first training results of the large language model of the first device for the input text data in the public text samples are combined, and the small language model with better natural understanding ability of text data is selected to transfer its learned knowledge to the large language model of the first device. As a result, the first device can optimize the large language model based on the knowledge transferred by the small language model and improve the natural language processing ability of the large language model.

[0107] In a feasible implementation, a specific method for performing a quality comparison analysis on the first training result and the second training result corresponding to each second device is proposed. Specifically, the prediction quality of the output text prediction results predicted by different devices can be compared based on a preset question-answering model. Among them, the second training result for the input text data generated based on the small language model of the second device may include the second output text prediction result. Similarly, the first training result for the input text data generated by the large language model of the first device may include the first output text prediction result. Accordingly, the above step S13 may include: generating a first training result including the first output text prediction result.

[0108] The above step S201 may include step S2011:

[0109] Step S2011, input the input text data, the first output text prediction result in the first training result, and the second output text prediction results in each second training result into a preset question-answering model, so that the question-answering model evaluates the matching degree between the first output text prediction result, each second output text prediction result and the input text data, and outputs a first comparison result based on the matching degree.

[0110] In the present embodiment, the question-answering model may select an independent open source and trained third-party question-answering model, which is used to evaluate the matching degree between the input text data and each second output text prediction result based on the input text data and multiple candidate second output text prediction results, and determine the accuracy of each second output text prediction result based on the matching degree (that is, the prediction effect of the second device on the output text corresponding to the input text data), thereby determining the quality of each second training result based on the accuracy of each second output text prediction result. Alternatively, the question-answering model for the sorting task can be specially trained. Specifically, the second device can use the labeled sample data to train the pre-built machine learning model, and optimize the performance of the model on the sorting task by adjusting the parameters of the model. Among them, the labeled sample data may include text (such as a question or sentence) and a corresponding label, for example, in the form of (<metric_response_1,metric_response_2,…> , which one is the best). It should be noted that this embodiment does not limit the specific implementation method of the question-answering model.

[0111] In another feasible implementation, another specific method for quality comparison and analysis of the first training result and the second training result corresponding to each second device is proposed. Specifically, the first device can calculate the prediction effect measurement index value of the large language model on the input text data (hereinafter referred to as the "first prediction effect measurement index value") under the preset prediction effect measurement index, and calculate the prediction effect measurement index value of the small language model of each second device on the input text data (hereinafter referred to as the "second prediction effect measurement index value"), and evaluate and compare the quality of each second training result through the quantified index value. Therefore, in this implementation, the second training result includes the second prediction effect measurement index value, which is the index value calculated by the second device based on the second output text prediction result and the label text data in the public text sample; accordingly, the above step S13 may include: based on the first output text prediction result and the label text data in the public text sample, calculate the measurement index value of the prediction effect of the large language model on the input text data to obtain the first prediction effect measurement index value; generate the first training result including the first prediction effect measurement index value.

[0112] The above step S201 may include step S2011:

[0113] S2011: Compare the first prediction effect measurement index value in the first training result and the second prediction effect measurement index value in each second training result according to the index value to obtain a first comparison result.

[0114] Exemplarily, the first device may use prediction effect metrics such as cross entropy, sample accuracy, etc. to calculate the metric values ​​of the prediction effects of the large language model and the small language model on the input text data. When the first device obtains the first prediction effect metric value of the large language model for the input text data and the second prediction effect metric value of each small language model for the input text data, the first device may sort the first prediction effect metric value and each second prediction effect metric according to the size of the metric value, and determine the first training result and the quality sorting result of each second device based on the sorting result of the metric value, and the obtained first training result and the quality sorting result of each second device are the first comparison result.

[0115] In a feasible implementation, in order to further optimize the traffic received by the first device, in this implementation, for each public text sample, at most one device is set to which the public text sample belongs. Specifically, the above step S202 may include steps S2021 to S2022:

[0116] Step S2021, for each public text sample, based on the first comparison result corresponding to the public text sample, determine the optimal training result corresponding to the input text data in the public text sample, the optimal training result being the training result with the best quality among the first training result corresponding to the input text data and all the corresponding second training results.

[0117] After the first device performs the operation of the above step S201 for each public text sample, it can obtain the first comparison result corresponding to each public text sample. The first comparison result corresponding to each public text sample can include the optimal training result corresponding to the input text data in the corresponding public text sample, or can also include the quality ranking result of the training result corresponding to the input text data. Thus, the first device can determine the optimal training result corresponding to the input text data in each public text sample based on the first comparison result corresponding to each public text sample, thereby determining the language model with the best natural language processing capability for each public text data (including the large language model of the first device and the small language models of each second device). The optimal training result can be the first training result or any second training result.

[0118] Step S2021, taking the second training result that meets the preset condition as the target second training result: the preset condition is: the second training result is the optimal training result corresponding to the input text data.

[0119] When the optimal training result corresponding to the input text data in any public text sample is one of the second training results, it is determined that the second training result meets the preset condition. It can be understood that in this case, the small language model of the target second device corresponding to the second training result has the best natural language processing capability for the public text sample. Thus, the first device selects the small language model so that the small language model transfers the knowledge learned for the specific input text data.

[0120] Through the above solution, the large language model of the first device can learn specific knowledge with high accuracy in a certain niche learned by the small language model, thereby significantly improving the natural language processing capability of the large language model.

[0121] Based on the above first embodiment and / or second embodiment, a third embodiment of the language model training method of the present application is proposed. Figure 2 , Figure 2This is a flow chart of the third embodiment of the language model training method of the present application. In this embodiment, the same or similar contents as those of the first and second embodiments can be referred to the above description, and will not be repeated later. In this embodiment, the language model training method is applied to any one of a plurality of second devices, that is, the execution subject is any second device, and the method includes steps A10 to A70:

[0122] Step A10: Use the local private text sample to train the small language model, and predict the input text data in the public text sample based on the trained small language model to obtain a second output text prediction result and a second text probability distribution prediction result.

[0123] Step A20: Generate a second training result based on the second output text prediction result.

[0124] Specifically, after obtaining the second output text prediction result, the second device can use the second output text prediction result as the second training result, or the second device can calculate the metric value of the prediction effect of the small language model of the second device on the input text data based on the second output text prediction result and the label text data in the public text sample, obtain the second prediction effect metric value, and use the second prediction effect metric value as the second training result, or the second device can set the second prediction result to include the second output text prediction result and the second prediction effect metric value.

[0125] Step A30, sending the second training result to the first device, so that after the first device determines the target second training result and the corresponding target second device that meet the preset conditions, it sends the first sending indication information to each second device, the public text sample belongs to the target second device, and the first sending indication information is used to instruct the target second device to send the second text probability distribution prediction result of the input text data in the public text sample to the first device, and the second text probability distribution prediction result is the result obtained by the target second device inputting the input text data into the trained small language model for prediction.

[0126] The number of target second devices is less than the number of second devices participating in the training.

[0127] Step A40, upon receiving the first sending indication information sent by the first device, sending the second text probability distribution prediction result of the input text data to the first device, so that the first device trains a large language model based on the second text probability distribution prediction result.

[0128] Specifically, the second device transmits the second text probability distribution prediction result corresponding to the input text data to the first device only when it receives the first sending indication information sent by the first device. Therefore, compared with the traditional method in which each second device needs to transmit the text probability distribution information of each public text sample, this embodiment can reduce the communication volume transmitted between the first device and the second device, and can correspondingly reduce the amount of data received by the first device in a short time, thereby alleviating the load pressure on the first device.

[0129] Step A50, receiving the third training result sent by the first device, the third training result is a training result generated based on the third output text prediction result after the first device inputs the input text data in the public text sample into the trained large language model for prediction to obtain the third output text prediction result.

[0130] Specifically, after the first device optimizes the large language model based on the first text probability distribution prediction result sent by the second device, the first device inputs the input text data in the public text sample into the trained large language model for prediction to obtain a third output text prediction result, generates a third training result based on the third output text prediction result, and sends the third training result to each second device. The second device receives the third training result to determine the natural language processing capability of the trained large language model of the first device for the input text data based on the third training result.

[0131] Step A60, performing a comparative analysis on the training quality of the second training result and the third training result, and when determining that the quality of the third training result is better than the quality of the second training result, sending a second sending indication information to the first device, the second sending indication information is used to instruct the first device to send a first text probability distribution prediction result of the input text data to the second device, the first text probability distribution prediction result is a result obtained by the first device inputting the input text data into the trained large language model for prediction.

[0132] Specifically, the second device can use a preset quality assessment strategy to compare the quality of the second training result and the third training result, so as to determine a language model with stronger learning and understanding capabilities for the input text data in the public text sample. When the second device determines that the quality of the first training result of the large language model for the input text data is better than the quality of the second training result of the local small language model of the second device for the input text data, it determines that the natural language understanding capability of the large language model of the first device for the input text data is better than the natural language understanding capability of the local small language model for the input text data. At this time, the second device sends a second sending indication message to the first device, so that the first device, when receiving the second sending indication message, sends to the second device the text probability distribution information learned by the large language model deployed by it for the input text data. In some embodiments, the second device may send second sending indication information via a second notification message. The second notification message may be in the form of a vector, and the size of the vector is N*1, where N is the number of public text samples. The i-th variable in the vector represents the second indication information of the i-th public text sample. When the value of the i-th variable is 1, it represents the second sending indication information for the i-th public text sample, which is used to indicate that the first device needs to send a first text probability distribution prediction result for the input text data in the public text sample predicted by the large language model of the first device. When the value of the i-th variable is 0, it represents the second static indication information for the i-th public text sample, which is used to indicate that the first device does not need to send the first text probability distribution prediction result for the input text data in the public text sample predicted by the large language model of the first device.

[0133] Step A70: receiving a first text probability distribution prediction result sent by the first device, and optimizing the small language model based on the first text probability distribution prediction result.

[0134] If the preset training end condition is met, step A80 is executed: obtaining a trained small language model.

[0135] If the preset training end condition is not met, the process returns to step A10.

[0136] After receiving the first text probability distribution prediction result sent by the first device, the second device trains a local small model to fit the first text probability distribution prediction result, so that the small language model of the second device can learn the text knowledge information transmitted by the large language model.

[0137] It should be noted that in the existing federated large and small model collaborative fine-tuning algorithm, the first device usually needs to send the text probability distribution information learned for each input text data to all second devices. When the number of public text samples is huge, the amount of data that the second device needs to process in a short period of time increases significantly, especially considering that these devices often face relatively tight computing resources in enterprise deployment. In this embodiment, by comparing the quality of the training results of the large language model on the input text data with the training results of the small language model on the input text data, the first device will only pass the text probability distribution information of the input text data to the second device only when specific conditions are met (the quality of the training results of the large language model on the input text data is better than the quality of the training results of the small language model based on the input text data), which can effectively reduce the amount of communication received by the second device in specific scenarios.

[0138] After the second device trains the small language model based on the second text probability distribution prediction result, it can detect whether the preset training end condition is met. The preset training end condition is a condition that needs to be met to end the pre-training training, which can be set as needed, and is not limited in this embodiment. For example, in a specific implementation, it can be set to the convergence of the loss function of the small language model or the number of training rounds reaching the set number of rounds, or the training duration reaching the set duration, etc.

[0139] If the second device detects that the preset training end condition is met, the iterative training process can be ended, and the finally updated small language model is used as the trained small language model.

[0140] In this embodiment, by comparing the third prediction result of the large language model for the input text data with the second training result of the small language model for the input text data, it is determined whether the first device needs to transmit the text probability distribution information of the input text data. Therefore, this embodiment can also achieve the purpose of reducing the amount of communication received by the first device in some scenarios.

[0141] In a feasible implementation, a solution is proposed in which the second device trains a small language model when the second training result is better than the third training result. Specifically, the above step A60 may further include step A70:

[0142] Step A70: When it is determined that the quality of the second training result is better than the quality of the third training result, optimize the small language model based on the second text probability distribution prediction result.

[0143] Specifically, when the second device determines that the second training result is better than the third training result, it determines that the natural language understanding ability of the local small language model for the input text data in the public text sample is better than the natural language understanding ability of the large language model of the first device for the input text data. Therefore, the parameters of the small language model are updated directly based on the first text probability distribution prediction result obtained by the local large language model for the input text data.

[0144] In a feasible implementation, a specific method for comparing the quality of the second training result and the third training result is proposed. Specifically, the third training result may include a third output text prediction result. Accordingly, the step A20 may include: generating a third training result including the third output text prediction result;

[0145] Accordingly, in the above step A60, comparing the quality of the second training result and the third training result may include: inputting the input text data, the second output text prediction result in the second training result, and the third output text prediction result in the third training result into a preset question-answering model, so that the question-answering model evaluates and outputs a training result with the best training quality for the input text data.

[0146] The specific implementation of the above step A60 can refer to the specific implementation of step S2011 in the above second embodiment. The implementation principles of the two are the same and will not be elaborated here.

[0147] In another feasible implementation, another specific method for comparing the quality of the second training result and the third training result is proposed. Specifically, the third prediction effect metric index value of the third training result is a metric index value of the prediction effect of the large language model of the first device on the input text data, calculated by the first device based on the third output text prediction result and the label text data in the public text sample. Accordingly, the above step A20 may include: based on the second output text prediction result and the label text data in the public text sample, calculating the metric index value of the prediction effect of the small language model of the second device on the input text data to obtain a second prediction effect metric index value; generating a second training result including the second prediction effect metric index value;

[0148] In the above step A60, comparing the second training result with the third training result may include: comparing the second prediction effect metric index value in the second training result with the third prediction effect metric index value in the third training result according to the magnitude of the prediction loss value.

[0149] The specific implementation of the above step A60 can refer to the specific implementation of step S2011 in the above second embodiment. The implementation principles of the two are the same and will not be elaborated here.

[0150] In order to help understand the above embodiments of the language model training method of the present application, an example of a language model training method combined with the above embodiments is proposed. Figure 3 , the first device can be called a federated server, and each second device can be called a client. The large language model LLM of the server can adopt the LLaMa213B model, and the small language model SLM of each client can adopt, for example, GPT-2, OPT 7B, LLaMa2 7B model. Privatedata represents the private text samples of each client, and public data represents the public text samples. Adapter modules are attached to the pre-trained large language model (LLM) and small language model (SLM) respectively.

[0151] During each round of collaborative fine-tuning, each client uses local private text sample private data to train the small language model, and then calculates the second training results obtained after the trained small language model is trained on the input text data in each public text sample. The second training result can be the second output text prediction result obtained by the client inputting the input text data in the public text sample into the trained small language model for prediction, or it can be the result obtained by the client calculating the difference (such as cross entropy) between the second output text prediction result and the label text data in the public text sample after obtaining the second output text prediction result. After obtaining the second training results corresponding to the input text data in each public text sample, each client uploads the second training results corresponding to the input text data in each public text sample to the server. Please refer to . Figure 3 , Figure 3 A schematic diagram of an application scenario involved in the language model training process is given. Figure 3 As shown, client A generates a second training result A, client B generates a second training result B, and client C generates a second training result C and sends it to the server.

[0152] The server calculates the first training result of the large language model for each public text sample in the public sample set, and then executes the following process: For details, see Figure 4 , Figure 4 A schematic diagram of another application scenario involved in the language model training process is given. Figure 4As shown, for each public text sample, the server aggregates the second training results sent by all clients corresponding to the public text sample and the first training results corresponding to the server. The format of the aggregated training result of each public text sample (also called aggregated training result) is (first training result, second training result A, second training result B, second training result C, ...), and the aggregated training result (first training result, second training result A, second training result B, second training result C, ...) corresponding to each public text sample is respectively input into the sample quality evaluator Evaluator, and the target training result (also called "optimal training result", hereinafter collectively referred to as "optimal target training result") and the device corresponding to the optimal training result (hereinafter collectively referred to as "optimal device") uniquely corresponding to each public text sample are determined by the sample quality evaluator Evaluator. The input text sample belongs to the optimal device, and the optimal device can be the server or any client.

[0153] The server sends a first notification message corresponding to each client to each client, the first notification message includes first indication information for each public text sample, the first indication information is used to indicate whether the client needs to send a text probability distribution prediction result of any public text sample to the server, the first notification information can be in the form of a vector, and the size of the vector is N*1, N is the number of public text samples, the i-th variable in the vector corresponds to the first indication information of the public text sample with index number i, the value of the i-th variable in the vector is 0 or 1, wherein 0 indicates that the optimal device to which the public text sample with index number i belongs is not the client, and the client does not need to send the second text probability distribution prediction result for the input text data in the public text sample predicted by the client's small language model to the server, and 1 indicates that the optimal device to which the public text sample with index number i belongs is the client, and the client needs to send the second text probability distribution prediction result for the input text data in the public text sample predicted by the client's small language model to the server.

[0154] For each client, the input text data in each public text sample is respectively input into the trained small language model to obtain the second text probability distribution prediction result corresponding to the input text data in each public text sample. After receiving the first notification message, if the client determines through the first notification message that any first indication information is the first sending indication information (that is, the value of the first indication information is 1), the second text probability distribution prediction result of the input text data in the public text sample corresponding to the indication is sent to the server.

[0155] After receiving the second text probability distribution prediction results of the input text data in the corresponding public text samples sent by each client, the server trains a large model to fit the second text probability distribution prediction results of the input text data in the corresponding public text samples to obtain a trained large language model.

[0156] The server uses the trained large language model to generate a training result for the input text data in each public text sample, also called a third training result, and sends the third training result corresponding to the input text data in each public text sample to each client.

[0157] After receiving the third training result of the input text data in each public text sample sent by the server, each client combines the third training result and the second training result of the input text data in each public text sample into the sample quality evaluator Evaluator based on the sample identifier or sample index of the public text sample. The sample quality evaluator Evaluator outputs the best training result of the third training result and the second training result, and sends a second notification message corresponding to each client to each client, the second notification message includes second indication information for each public text sample, and the second indication information is used to indicate whether the server needs The text probability distribution prediction result of any public text sample is sent to the client, wherein the second notification message is in the form of a vector, the size of the vector is N*1, N is the number of public text samples, the i-th variable in the vector corresponds to the public text sample with sample index i, and the value of the i-th variable in the vector is 0 or 1, wherein 0 indicates that the optimal device to which the public text sample with index number i belongs is not the server, and 1 indicates that the optimal device to which the public text sample with index number i belongs is the server, and the server needs to send to the client the first text probability distribution prediction result for the input text data in the public text sample obtained by predicting the large language model deployed by it.

[0158] The server inputs the input text data in each public text sample into the trained large language model to obtain the first text probability distribution prediction results corresponding to the input text data in each public text sample.

[0159] After receiving the second notification message corresponding to any client, the server determines whether it is necessary to send the text probability distribution prediction result of any public text sample based on the second indication information contained in the second notification message. If the value of the second indication information of any public text sample is 1, the server determines that it has received the second sending indication information of the public text sample, and changes the value of the second sending indication information corresponding to the public text sample to the first text probability distribution prediction result of the input text data in the public text sample, and obtains a second response result to feed back to the client.

[0160] After receiving the second response result sent by the server, each client optimizes the parameters of the large language model for the public text samples associated with the second indication information of 0 in the second response result based on the first text probability distribution prediction result predicted by the large language model for the input text data in the public text samples, and for the public text samples whose second indication information value is not 0, that is, the server sends the public text samples corresponding to the first text probability distribution result, the first text probability distribution result predicted by the large language model of the server is used to fit and train the local small language model.

[0161] Each round of collaborative fine-tuning is iterated until the preset training end conditions are met, and the trained large language model and small language model are obtained.

[0162] The following provides a specific implementation of the sample quality evaluator. Specifically, the training results (including the first training results, the second training results, and the third training results) can be preset prediction effect metrics, such as cross entropy, sample accuracy, and can also be the output text prediction results obtained by the corresponding language model predicting the input text data, that is, a response text.

[0163] For directly quantifiable prediction effect metrics such as cross entropy and sample accuracy, the training result with the best indicator value is selected as the optimal training result. For the case where the training result is the output text prediction result, the sample quality evaluator has the following two strategies:

[0164] Strategy a: No training is required. An independent open-source third-party large model is selected. Then, the output text prediction results to be compared are combined into a multiple-choice question and input into the third-party large model to select the best output text prediction result. A similar prompt format can be: "Given the original question Q, please select the best answer from the following options\n\nOption 1: {output text prediction result A}\n\n, Option 2: {output text prediction result B}..." The prompt can be adjusted and adapted according to the characteristics of the specific model, or the industry-wide few-shot method can be used to obtain better selection quality. The advantage of this method is that it does not require training, but the disadvantage is that it has relatively high requirements for the capabilities of the third-party model.

[0165] Strategy b: Artificially label the classification results (good / bad) or ranking results of some output text prediction result groups to be compared (each output text prediction result group includes at least two output text prediction results), and then specially train a scoring model as an Evaluator. The task type can be a classification task or a ranking task. For the classification task, the constructed sample can be (<output text prediction result A, output text prediction result B>, 0 / 1), indicating whether the output text prediction result A is better than the output text prediction result B, or the constructed sample can be directly (<output text prediction result A, output text prediction result B,…>, which one is the best).

[0166] Through the above solution, for the server, the amount of communication received by the server is reduced from client_num (number of clients) * epoch (number of iterations) * N (sample size) * seq_len (length of sentence after word segmentation) * vocab_size (vocab of small language model) floating point numbers to epoch (number of iterations) * N (sample size) * seq_len (length of sentence after word segmentation) * vocab_size (vocab). For the client, the amount of communication received by the client will be consistent with that before optimization in the worst case (that is, each sample of the client chooses to use the text probability distribution of the server for training).

[0167] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the language model training method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0168] An embodiment of the present application provides a language model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the language model training method in the above-mentioned embodiment 1.

[0169] Reference below Figure 5 , which shows a schematic diagram of the structure of a language model training device suitable for implementing the embodiment of the present application. The language model training device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The language model training device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0170] like Figure 5 As shown, the language model training device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the language model training device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the language model training device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a language model training device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0171] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0172] Compared with the prior art, the beneficial effects of the language model training device provided in the embodiment of the present application are the same as the beneficial effects of the language model training method provided in the above embodiment, and other technical features in the language model training device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0173] It should be understood that the various parts disclosed in the embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0174] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0175] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, and the computer-readable program instructions are used to execute the language model training method in the above-mentioned embodiment.

[0176] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0177] The computer-readable storage medium may be included in the language model training device; or may exist independently without being assembled into the language model training device.

[0178] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the language model training device, the language model training device performs the above-mentioned functions defined in the method of the embodiment disclosed in this application.

[0179] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0182] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned language model training method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the language model training method provided in the above-mentioned embodiment, which will not be repeated here.

[0183] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned language model training method when executed by a processor.

[0184] Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present application are the same as the beneficial effects of the language model training method provided in the above embodiments, which will not be elaborated here.

[0185] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A language model training method, characterized in that: Applied to a first device, the first device is respectively connected to each second device for communication, a large language model to be trained is deployed in the first device, and a small language model to be trained is deployed in each second device, the method includes: receiving a second training result respectively sent by each of the second devices, where the second training result is a training result generated by the second device based on a second output text prediction result, where the second output text prediction result is a result obtained by the second device training the small language model using a local private text sample, and then inputting input text data in a public text sample into the trained small language model for prediction; Determine a target second training result and a corresponding target second device that meet a preset condition, the public text sample belongs to the target second device, and the number of the target second devices is less than the number of second devices participating in the training; Sending first sending indication information to the target second device, wherein the first sending indication information is used to instruct the target second device to send a second text probability distribution prediction result of the input text data in the public text sample to the first device, wherein the second text probability distribution prediction result is a result obtained by the target second device inputting the input text data into the trained small language model for prediction; Receiving the second text probability distribution prediction result sent by the target second device, and training the large language model based on the second text probability distribution prediction result; Return to the above step of receiving the second training results sent by each of the second devices respectively, until the preset training end condition is met, and obtain a large language model that has completed the training.

2. The method according to claim 1, characterized in that Before the step of determining the target second training result and the corresponding target second device that meet the preset conditions, the following step is also included: Inputting input text data in the public text sample held by the first device into the large language model for prediction to obtain a first output text prediction result; Generate a first training result based on the first output text prediction result; The step of determining a target second training result and a corresponding target second device that meet a preset condition comprises: Performing a comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result; Selecting a target second training result that meets a preset condition based on the first comparison result, wherein the target second training result is better than the first training result in quality; Determine that the second device corresponding to the target second training result is the target second device.

3. The method according to claim 2, characterized in that The public text samples include a plurality of pieces; the step of selecting a target second training result that meets a preset condition based on the first comparison result includes: For each of the public text samples, based on the first comparison result corresponding to the public text sample, determining the optimal training result corresponding to the input text data in the public text sample, the optimal training result being the training result with the best quality among the first training result corresponding to the input text data and all the corresponding second training results; The second training result that meets a preset condition is used as the target second training result, and the preset condition is that the second training result is the optimal training result corresponding to the input text data.

4. The method according to claim 2, characterized in that The second training result includes the second output text prediction result; and the step of generating the first training result based on the first output text prediction result includes: Generate a first training result including the first output text prediction result; The step of performing comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result includes: The input text data, the first output text prediction result in the first training result, and the second output text prediction results in each of the second training results are input into a preset question-answering model, so that the question-answering model evaluates the matching degree between the first output text prediction result, each of the second output text prediction results and the input text data, and outputs the first comparison result based on the matching degree.

5. The method according to claim 2, characterized in that The second training result includes a second prediction effect metric index value, where the second prediction effect metric index value is a metric index value of a prediction effect of the small language model of the second device on the input text data, calculated by the second device based on the second output text prediction result and the label text data in the public text sample; The step of generating a first training result based on the first output text prediction result comprises: Based on the first output text prediction result and the label text data in the public text sample, calculating a metric value of the prediction effect of the large language model on the input text data to obtain a first prediction effect metric value; Generate a first training result including the first prediction effect measurement indicator value; The step of performing comparative analysis on the training quality of the first training result and the second training results corresponding to each of the second devices to obtain a first comparative result includes: The first prediction effect measurement index value in the first training result and the second prediction effect measurement index value in each of the second training results are compared according to the size of the index value to obtain the first comparison result.

6. A language model training method, characterized in that: Applied to any second device among a plurality of second devices, each of the second devices is respectively connected to a first device for communication, a large language model to be trained is deployed in the first device, and a small language model to be trained is respectively deployed in each of the second devices, the method comprising: Using a local private text sample to train the small language model, and based on the trained small language model, predicting the input text data in the public text sample to obtain a second output text prediction result and a second text probability distribution prediction result; Generate a second training result based on the second output text prediction result; The second training result is sent to the first device, so that after the first device determines the target second training result and the corresponding target second device that meet the preset conditions, the first device sends a first sending indication information to each second device, the public text sample belongs to the target second device, the number of the target second devices is less than the number of second devices participating in the training, and the first sending indication information is used to instruct the target second device to send the second text probability distribution prediction result of the input text data in the public text sample to the first device, and the second text probability distribution prediction result is the result obtained by the target second device inputting the input text data into the trained small language model for prediction; Upon receiving the first sending indication information sent by the first device, sending a second text probability distribution prediction result of the input text data to the first device, so that the first device trains the large language model based on the second text probability distribution prediction result; receiving a third training result sent by the first device, wherein the third training result is a training result generated based on the third output text prediction result after the first device inputs the input text data in the public text sample into the trained large language model for prediction to obtain a third output text prediction result; Performing a comparative analysis on the training quality of the second training result and the third training result, and in the case where it is determined that the quality of the third training result is better than the quality of the second training result, sending second sending indication information to the first device, wherein the second sending indication information is used to instruct the first device to send a first text probability distribution prediction result of the input text data to the second device, wherein the first text probability distribution prediction result is a result obtained by the first device inputting the input text data into the trained large language model for prediction; Receiving the first text probability distribution prediction result sent by the first device, and optimizing the small language model based on the first text probability distribution prediction result; Return to the above step of training the small language model using local private text samples until the preset training end condition is met, thereby obtaining a trained small language model.

7. The method according to claim 6, characterized in that After the step of performing comparative analysis on the training quality of the second training result and the third training result, the method further includes: In a case where it is determined that the quality of the second training result is better than the quality of the third training result, the small language model is optimized based on the second text probability distribution prediction result.

8. The method according to claim 6, characterized in that The third training result includes the third output text prediction result; The step of generating a second training result based on the second output text prediction result comprises: Generate a second training result including the second output text prediction result; The step of performing comparative analysis on the training quality of the second training result and the third training result comprises: The input text data, the second output text prediction result in the second training result, and the third output text prediction result in the third training result are input into a preset question-answering model, so that the question-answering model evaluates and outputs a training result with the best training quality for the input text data.

9. The method according to claim 6, characterized in that The third training result includes a third prediction effect metric index value, where the third prediction effect metric index value is a metric index value of a prediction effect of the large language model of the first device on the input text data, calculated by the first device based on the third output text prediction result and the label text data in the public text sample; The step of generating a second training result based on the second output text prediction result comprises: Based on the second output text prediction result and the label text data in the public text sample, calculating a metric value of the prediction effect of the small language model of the second device on the input text data to obtain a second prediction effect metric value; Generate a second training result including the second prediction effect metric value; The step of performing comparative analysis on the training quality of the second training result and the third training result comprises: The second prediction effect measurement index value in the second training result and the third prediction effect measurement index value in the third training result are compared according to the size of the index value.

10. A language model training device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the language model training method according to any one of claims 1 to 9.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 9 are implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the language model training method according to any one of claims 1 to 9 are implemented.

Images