Model training method and related product

Abstract

The invention discloses a model training method and a related product. The method comprises the following steps: deploying a first model in a running memory of i devices; deploying a second model in the running memory of the m devices; deploying a first model in video memories of n devices; processing the training prompt word by using a first model in the n devices to generate a first text; transferring the first models in the video memories of the n devices from the video memories to a running memory; transferring the first model deployed in the i devices from a running memory to a video memory; transferring the second model deployed in the m devices from the running memory to the video memory; processing the training prompt word and the first text by using a second model in a video memory of m devices, and estimating a first improvement degree of a target word in the first text to a target matching degree; determining a first loss of the first model based on the first lifting degree; and based on the first loss, updating parameters of the first model in the video memory of the i devices to obtain a target model.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a model training method and related products. Background Art

[0002] With the development of natural language processing technology, large language models (LLMs) have emerged. Thanks to their powerful performance, LLMs are widely used. However, before LLMs are put into application scenarios, they need to be trained. Therefore, how to train LLMs is of great significance. Summary of the invention

[0003] The present application provides a model training method and related products, wherein the related products include a model training device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a model training method is provided, the method comprising:

[0005] Deploy a first model in the running memory of i devices, where the first model is a model with natural language processing capability;

[0006] Deploy a second model in the running memory of the m devices, where the second model is used to estimate the degree of improvement of the generated words to a target matching degree, where the target matching degree is the matching degree between the text generated by the first model and a preset requirement;

[0007] Deploy the first model in the video memory of n devices, where n is the sum of i and m;

[0008] Using the first model deployed in the video memory of the n devices, the training prompt words are processed to generate a first text;

[0009] Transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices;

[0010] Transferring the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0011] Transferring the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0012] Using the second model in the video memory of the m devices, the training prompt word and the first text are processed to estimate a first improvement degree of the target word in the first text to the target matching degree;

[0013] determining a first loss of the first model based on the first lift, wherein the first lift is negatively correlated with the first loss;

[0014] Based on the first loss, the parameters of the first model in the video memory of the i devices are updated to obtain a target model.

[0015] In combination with any implementation manner of the present application, the deploying the first model in the video memory of n devices includes:

[0016] Divide the first model into n parts to obtain n first sub-models;

[0017] Deploy the n copies of the first sub-model in the video memory of the n devices, wherein the first sub-model corresponds to a device in the n devices one by one;

[0018] The step of processing the training prompt words by using the first model deployed in the video memory of the n devices to generate the first text includes:

[0019] The training prompt words are processed by using the n first sub-models deployed in the video memory of the n devices to generate the first text.

[0020] In combination with any implementation manner of the present application, the n devices include a shared device, the video memory of the shared device includes a shared storage space, the n devices can access the shared storage space, and the n first sub-models are stored in the shared storage space;

[0021] The step of processing the training prompt words by using the n first sub-models deployed in the video memory of the n devices to generate the first text includes:

[0022] For each of the n devices, target data required for completing target processing of the corresponding first sub-model is obtained from the shared storage space, where the target processing is used to implement processing of the training prompt word by the first model;

[0023] The target processing is performed on the corresponding target data using the n first sub-models deployed in the video memories of the n devices to generate the first text.

[0024] In combination with any implementation manner of the present application, for each of the n devices, obtaining target data required to complete target processing of the corresponding first sub-model from the shared storage space includes:

[0025] For each of the n devices, determining data required to complete the target processing of the corresponding first sub-model from the shared storage space as the target data;

[0026] For each of the n devices, the target data stored in the shared storage space is stored in a video memory of the device.

[0027] In combination with any implementation manner of the present application, the deploying the first model in the running memory of i devices includes:

[0028] Divide the first model into i parts to obtain i second sub-models;

[0029] Deploy the i copies of the second sub-model in the running memory of the i devices, wherein the second sub-model corresponds one-to-one to the devices in the i devices;

[0030] The transferring the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices includes:

[0031] Transferring the i copies of the second sub-model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0032] The updating, based on the first loss, the parameters of the first model in the video memory of the i devices to obtain the target model includes:

[0033] Based on the first loss, update the parameters of the i second sub-models in the video memory of the i devices to obtain i third sub-models;

[0034] The i third sub-models are merged to obtain the target model.

[0035] In combination with any embodiment of the present application, the step of transferring the i second sub-models deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices includes:

[0036] The first training information stored in the running memory of the i devices is transferred to the video memory of the i devices, wherein the first training information includes: parameters of the second sub-model, the gradient of the first model in the previous round of training, and the current number of training rounds.

[0037] In combination with any embodiment of the present application, the i third sub-models are obtained through training of the current number of training rounds;

[0038] The merging of the i third sub-models to obtain the target model includes:

[0039] When the current training round number is the last training round number, the i third sub-models are merged to obtain the target model.

[0040] In combination with any embodiment of the present application, the method further includes:

[0041] When the current training round number is not the last training round number, transferring the i copies of the third sub-model from the video memory of the i devices to the running memory of the i devices;

[0042] Each of the n devices performs the step of merging the i copies of the third sub-model in the running memory of the i devices, so that the running memory of each of the n devices stores the updated first model, and the updated first model is obtained by merging the i copies of the third sub-model;

[0043] Each of the n devices executes the step of dividing the updated first model into n fourth sub-models, so that a copy of the fourth sub-model is stored in the running memory of each of the n devices, and the fourth sub-models stored in the running memories of different devices among the n devices are different;

[0044] Each of the n devices executes to store the fourth sub-model stored in the running memory into the shared storage space.

[0045] In combination with any implementation manner of the present application, the deploying the second model in the running memory of the m devices includes:

[0046] Divide the second model into m parts to obtain m fifth sub-models;

[0047] Deploy the m copies of the fifth sub-model in the running memory of the m devices, wherein the fifth sub-model corresponds one-to-one to a device in the m devices;

[0048] The transferring the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices includes:

[0049] Transferring the m copies of the fifth sub-model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0050] The using the second model in the video memory of the m devices to process the training prompt word and the first text, and estimating and generating a first improvement degree of the target word in the first text to the target matching degree, includes:

[0051] The m fifth sub-models in the video memory of the m devices are used to process the training prompt words and the first text to estimate the first lift.

[0052] In combination with any implementation manner of the present application, the step of transferring the m copies of the fifth sub-model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices includes:

[0053] The second training information stored in the running memory of the m devices is transferred to the video memory of the m devices, wherein the second training information includes: parameters of the fifth sub-model, the gradient of the second model in the previous round of training, the current number of training rounds, and the first training data required to be processed by the fifth sub-model in the current number of training rounds, wherein the first training data is obtained by segmenting the training prompt words and the first text.

[0054] In combination with any implementation manner of the present application, a third model is deployed in the video memory of the target device among the n devices, and the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0055] After transferring the m copies of the fifth sub-models deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices, the method further includes:

[0056] Obtaining a second lift, where the second lift is an actual lift of the target word to the target matching degree, and the second lift is obtained by processing the training prompt word and the first text by the third model;

[0057] determining a second loss of the second model based on a first difference between the second lift and the first lift, wherein the second loss is positively correlated with the first difference;

[0058] Based on the second loss, update the parameters of the m fifth sub-models in the video memory of the m devices to obtain m sixth sub-models;

[0059] The m sixth sub-models are merged to obtain an updated second model.

[0060] In combination with any implementation manner of the present application, before transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices, the method further includes:

[0061] Deploy a third model in the running memory of a target device among the n devices, wherein the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0062] After transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices, and before determining the first loss of the first model based on the first improvement, the method further includes:

[0063] Transferring the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device;

[0064] Using the third model deployed in the video memory of the target device to process the training prompt word and the first text, a second lift is obtained, where the second lift is an actual lift of the target word to the target matching degree;

[0065] Using the second model in the video memory of the m devices, the training prompt word and the first text are processed to estimate a third improvement degree of a reference word in the first text to the target matching degree, wherein the reference word is a word preceding the target word;

[0066] The determining a first loss of the first model based on the first lift includes:

[0067] Determine an advantage function based on a second difference between the second lift and the third lift, the advantage function representing the improvement of the target matching degree when the target word is generated as the next word of the reference word when the reference word has been generated;

[0068] Based on the advantage function, the first loss is determined, the first loss being positively correlated with the first difference.

[0069] In a second aspect, a model training device is provided, the model training device comprising:

[0070] A deployment unit, configured to deploy a first model in a running memory of i devices, wherein the first model is a model having a natural language processing capability;

[0071] A deployment unit, configured to deploy a second model in the running memory of the m devices, wherein the second model is used to estimate the degree of improvement of the generated words to a target matching degree, wherein the target matching degree is the matching degree between the text generated by the first model and a preset requirement;

[0072] The deployment unit is used to deploy the first model in the video memory of n devices, where n is the sum of i and m;

[0073] A processing unit, configured to process the training prompt words using the first model deployed in the video memory of the n devices to generate a first text;

[0074] A transfer unit, configured to transfer the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices;

[0075] The transfer unit is used to transfer the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0076] The transfer unit is used to transfer the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0077] an estimating unit, configured to process the training prompt words and the first text using the second model in the video memory of the m devices, and estimate a first improvement degree of the target word in the first text to the target matching degree;

[0078] a loss determining unit, configured to determine a first loss of the first model based on the first lift, wherein the first lift is negatively correlated with the first loss;

[0079] An updating unit is used to update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model.

[0080] In conjunction with any embodiment of the present application, the deployment unit is specifically used to:

[0081] Divide the first model into n parts to obtain n first sub-models;

[0082] Deploy the n copies of the first sub-model in the video memory of the n devices, wherein the first sub-model corresponds to a device in the n devices one by one;

[0083] The processing unit is specifically used for:

[0084] The training prompt words are processed by using the n first sub-models deployed in the video memory of the n devices to generate the first text.

[0085] In combination with any embodiment of the present application, the n devices include a shared device, the video memory of the shared device includes a shared storage space, the n devices can all access the shared storage space, and the n first sub-models are stored in the shared storage space;

[0086] The above processing unit is specifically used for:

[0087] For each of the n devices, target data required for completing target processing of the corresponding first sub-model is obtained from the shared storage space, where the target processing is used to implement processing of the training prompt word by the first model;

[0088] The target processing is performed on the corresponding target data using the n first sub-models deployed in the video memories of the n devices to generate the first text.

[0089] In conjunction with any embodiment of the present application, the processing unit is specifically used to:

[0090] For each of the n devices, determining data required to complete the target processing of the corresponding first sub-model from the shared storage space as the target data;

[0091] For each of the n devices, the target data stored in the shared storage space is stored in a video memory of the device.

[0092] In conjunction with any embodiment of the present application, the deployment unit is specifically used to:

[0093] Divide the first model into i parts to obtain i second sub-models;

[0094] Deploy the i copies of the second sub-model in the running memory of the i devices, wherein the second sub-model corresponds one-to-one to the devices in the i devices;

[0095] The transfer unit is specifically used for:

[0096] Transferring the i copies of the second sub-model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0097] The above-mentioned updating unit is specifically used for:

[0098] Based on the first loss, update the parameters of the i second sub-models in the video memory of the i devices to obtain i third sub-models;

[0099] The i third sub-models are merged to obtain the target model.

[0100] In combination with any embodiment of the present application, the transfer unit is further used for:

[0101] The first training information stored in the running memory of the i devices is transferred to the video memory of the i devices, wherein the first training information includes: parameters of the second sub-model, the gradient of the first model in the previous round of training, and the current number of training rounds.

[0102] In combination with any embodiment of the present application, the i third sub-models are obtained through training of the current number of training rounds;

[0103] The above-mentioned updating unit is specifically used for:

[0104] When the current training round number is the last training round number, the i third sub-models are merged to obtain the target model.

[0105] In combination with any embodiment of the present application, the updating unit is further used to:

[0106] When the current training round number is not the last training round number, transferring the i copies of the third sub-model from the video memory of the i devices to the running memory of the i devices;

[0107] Each of the n devices performs the step of merging the i copies of the third sub-model in the running memory of the i devices, so that the running memory of each of the n devices stores the updated first model, and the updated first model is obtained by merging the i copies of the third sub-model;

[0108] Each of the n devices executes the step of dividing the updated first model into n fourth sub-models, so that a copy of the fourth sub-model is stored in the running memory of each of the n devices, and the fourth sub-models stored in the running memories of different devices among the n devices are different;

[0109] Each of the n devices executes to store the fourth sub-model stored in the running memory into the shared storage space.

[0110] In conjunction with any embodiment of the present application, the deployment unit is specifically used to:

[0111] Divide the second model into m parts to obtain m fifth sub-models;

[0112] Deploy the m copies of the fifth sub-model in the running memory of the m devices, wherein the fifth sub-model corresponds one-to-one to a device in the m devices;

[0113] The transferring the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices includes:

[0114] Transferring the m copies of the fifth sub-model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0115] The processing unit is specifically used for:

[0116] The m fifth sub-models in the video memory of the m devices are used to process the training prompt words and the first text to estimate the first lift.

[0117] In combination with any embodiment of the present application, the transfer unit is further used for:

[0118] The second training information stored in the running memory of the m devices is transferred to the video memory of the m devices, wherein the second training information includes: parameters of the fifth sub-model, the gradient of the second model in the previous round of training, the current number of training rounds, and the first training data required to be processed by the fifth sub-model in the current number of training rounds, wherein the first training data is obtained by segmenting the training prompt words and the first text.

[0119] In combination with any embodiment of the present application, a third model is deployed in the video memory of the target device among the n devices, and the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0120] The above estimation unit is also used for:

[0121] Obtaining a second lift, where the second lift is an actual lift of the target word to the target matching degree, and the second lift is obtained by processing the training prompt word and the first text by the third model;

[0122] The above-mentioned loss determination unit is also used for:

[0123] determining a second loss of the second model based on a first difference between the second lift and the first lift, wherein the second loss is positively correlated with the first difference;

[0124] The above update unit is also used for:

[0125] Based on the second loss, update the parameters of the m fifth sub-models in the video memory of the m devices to obtain m sixth sub-models;

[0126] The m sixth sub-models are merged to obtain an updated second model.

[0127] In combination with any embodiment of the present application, the deployment unit is further used to:

[0128] Deploy a third model in the running memory of a target device among the n devices, wherein the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0129] The above transfer unit is also used for:

[0130] Transferring the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device;

[0131] The above estimation unit is also used for:

[0132] Using the third model deployed in the video memory of the target device to process the training prompt word and the first text, a second lift is obtained, where the second lift is an actual lift of the target word to the target matching degree;

[0133] Using the second model in the video memory of the m devices, the training prompt word and the first text are processed to estimate a third improvement degree of a reference word in the first text to the target matching degree, wherein the reference word is a word preceding the target word;

[0134] The above-mentioned loss determination unit is specifically used for:

[0135] Determine an advantage function based on a second difference between the second lift and the third lift, the advantage function representing the improvement of the target matching degree when the target word is generated as the next word of the reference word when the reference word has been generated;

[0136] Based on the advantage function, the first loss is determined, the first loss being positively correlated with the first difference.

[0137] According to a third aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments described above.

[0138] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments.

[0139] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the first aspect and any one of its embodiments.

[0140] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the above-mentioned first aspect and any embodiment thereof.

[0141] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0142] In the present application, the model training device deploys the first model in the running memory of i devices, the second model in the running memory of m devices, and the first model in the video memory of n devices, wherein the n devices include i devices and m devices. After the first model deployed in the video memory of n devices is used to process the training prompt word to generate the first text, the reasoning process of the first model is completed. Then the first model deployed in the video memory of n devices is transferred from the video memory of n devices to the running memory of n devices, thereby releasing the video memory in n devices. After the video memory in n devices is released, the first model deployed in the running memory of i devices is transferred from the running memory of i devices to the video memory of i devices, and the second model deployed in the running memory of m devices is transferred from the running memory of m devices to the video memory of m devices. Then, the second model in the video memory of m devices is used to process the training prompt word and the first text, and the first improvement degree of the target word in the first text is estimated. The first loss of the first model is determined based on the first improvement degree, and the first improvement degree is negatively correlated with the first loss. Based on the first loss, the parameters of the first model in the video memory of i devices are updated to obtain the target model, thereby completing the training of the first model. Based on the technical solution provided in the embodiment of the present application, the reasoning of the first model and the training of the first model can be completed by n devices, and because the reasoning process of the first model and the training process of the first model are not performed simultaneously, the technical solution can alternately perform the reasoning process and the training process through n devices, thereby improving the utilization rate of device resources, thereby reducing the device resources required to complete the reasoning of the first model and the training of the first model. BRIEF DESCRIPTION OF THE DRAWINGS

[0143] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0144] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0145] Figure 1 A distributed training system architecture diagram provided for an embodiment of the present application;

[0146] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0147] Figure 3 Another distributed training system architecture diagram provided for an embodiment of the present application;

[0148] Figure 4 A flowchart of a model training provided in an embodiment of the present application;

[0149] Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0150] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0151] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0152] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0153] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0154] Thanks to its powerful performance, LLM is increasingly used. For example, it can be used to implement intelligent question and answer, translate, and extract text summaries. Before applying LLM, LLM needs to be trained. The process of training LLM (hereinafter referred to as the target training process) includes the inference process (generate) of LLM and the training process of LLM. The inference process of LLM includes inputting prompt words (promt) into LLM so that LLM generates text. The training process of LLM refers to updating the parameters of LLM based on the text generated by LLM. Due to the complex structure and parameters of LLM, more video memory resources are required to run LLM, while the video memory resources of a single device are limited. Therefore, distributed training is usually used to complete the target training process. Please refer to Figure 1 , Figure 1 A distributed training system architecture diagram provided for an embodiment of the present application. Specifically, the distributed training system includes n devices, each device includes video memory and running memory, wherein the video memory is used to run LLM, specifically, running LLM includes LLM reasoning and LLM training, and the running memory is used to store LLM model parameters and data in the target training process. When using the distributed training system for LLM reasoning or LLM training, it is necessary to divide the LLM into two or more sub-models and deploy different sub-models on different devices, for example, Figure 1 It shows that after the large model (ie, LLM) is divided into n sub-models, sub-model 1 is deployed on device 1, sub-model 2 is deployed on device 2, ..., and sub-model n is deployed on device n. The operation of LLM is realized by running different sub-models in parallel on n devices, thereby improving the operation speed of LLM, thereby improving the speed of the target training process.

[0155] The embodiment of the present application provides a model training method to obtain LLM through training in a distributed training system. The execution subject of the embodiment of the present application is a model training device, wherein the model training device can be a scheduler of the distributed training system. Optionally, the model training device can be one of the following: a computer, a server.

[0156] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 2 , Figure 2 A flowchart of a model training method provided in an embodiment of the present application.

[0157] 201. Deploy the first model in the running memory of the i devices.

[0158] In the embodiment of the present application, the running memory of the device is the memory required when the program is running, and optionally, the running memory is a random access memory (RAM). The first model is a language model, and the language model is a model with the ability of natural language processing (NLP). In a possible implementation, the language model is obtained by training with text training data, wherein the text training data includes one or more of masked text and non-masked text, and the model with NLP ability obtained by training with text training data also includes the model to be trained below. Masked text is a text including a mask. When the text training data includes masked text, the language model is trained using the text training data, so that the language model can predict the content obscured by the mask in the masked text to obtain a first prediction result, and then the loss of the language model is determined based on the difference between the first prediction result and the ground truth (GT) of the mask. The parameters of the language model are updated based on the loss until the loss of the language model converges, and the training of the language model can be completed. The non-masked text is the text that does not include the mask. When the text training data does not include the non-masked text, the language model is trained using the text training data, so that the language model can perform natural language processing tasks using the non-masked text, thereby improving the natural language processing capability of the language model, wherein the natural language processing tasks include: summarizing the text summary, determining the order of sentences, and determining the answer to the question based on the text. Optionally, the first model is a large language model (LLM).

[0159] Optionally, before the target training process is performed, the first model is a supervised fine-tuning (SFT) model, that is, during the target training process, the initial state of the first model is an SFT model, wherein the SFT model is obtained by supervised fine-tuning the language model. Specifically, the SFT model is obtained by supervised fine-tuning the language model using the target training data, wherein the target training data is labeled data obtained based on the target task labeling. For example, if the target task is to answer questions, then the target training data can be prompt words with labels, wherein the prompt words include questions, and the labels include answers to the questions in the prompt words. For another example, if the target task is to extract summaries, then the target training data can be prompt words with labels, wherein the prompt words include text to be extracted from the summary, and the labels include summaries of the text in the prompt words. Using the target training data to supervise fine-tune the language model to obtain the SFT model can enable the SFT model to have the ability to perform the target task, and accordingly, the first model has the ability to perform the target task.

[0160] At this time, the purpose of the target training process is to train the first model so that the result output by the first model meets the preset conditions, specifically, the result obtained by the first model performing the target task meets the preset conditions. Optionally, the preset requirements include one or more of human preferences, sentence fluency, and semantic accuracy human preferences. In one possible implementation, the preset requirement is human preference, and the target matching degree is the matching degree of human preference of the text generated by the first model. Then the purpose of the target training process is to make the result output by the first model meet human preference, that is, to align the result output by the first model with human preference through the target training process to achieve human preference alignment. For example, human preference can be a moral code, then the purpose of the target training process is to make the result output by the first model meet the moral code. For another example, human preference is the expression in the legal field, then the purpose of the target training process is to make the result output by the first model meet the expression in the legal field. In another possible implementation, the preset requirement is sentence fluency, and the target matching degree is the matching degree of sentence fluency of the text generated by the first model. Then the purpose of the target training process is to make the result output by the first model meet sentence fluency.

[0161] Optionally, the running of the first model is completed in the video memory of the device, that is, the first model needs to be inferred and trained in the video memory, so the video memory capacity of i devices is greater than or equal to the minimum video memory capacity, where the minimum video memory capacity is the minimum video memory capacity required for the first model to perform calculations and training.

[0162] In a possible implementation, the model training device can use a pipeline parallelism (PP) segmentation method to segment the first model to obtain i second sub-models. After the first model is segmented into i second sub-models, the i second sub-models are stored in i devices respectively, and the second sub-models stored in any device are different. Optionally, the model training device segments the first model so that the model layers are assigned to different second sub-models. For example, the first model includes the following four model layers: a first model layer, a second model layer, a third model layer, and a fourth model layer. The first model is segmented according to the PP segmentation method to obtain a second sub-model a and a second sub-model b, wherein the second sub-model a includes the first model layer and the second model layer, and the second sub-model b includes the third model layer and the fourth model layer.

[0163] 202. Deploy the second model in the running memory of the m devices.

[0164] In an embodiment of the present application, the second model is a language model. The second model is used to estimate the degree of improvement of the target match of the generated word, wherein the generated word is any word in the text generated by the first model, and the target match is the degree of match between the text generated by the first model and the preset requirements. Optionally, the model of the second model is obtained by adding a linear layer (value-head) after the last layer of the first model, wherein the linear layer is used to convert the result output by the first model into the degree of improvement of the target match of the generated word.

[0165] Specifically, the process of generating text by the first model is to generate the first word in the text, then generate the second word in the text, then generate the kth word in the text, and finally generate the complete text. The second model can determine the degree to which each word in the text improves the matching degree between the final generated complete text and the preset requirements.

[0166] For example, the preset requirement is sentence fluency, and the text generated by the language model based on the prompt word is a1, where the first word in a1 is a11, and the second word in the text a1 is a12. When the second model receives a1, it can estimate the degree to which the fluency of the final generated text a1 is improved by the first word a11, and it can also estimate the degree to which the fluency of the final generated text a1 is improved by the second word a12 when the first word is already a11.

[0167] In an embodiment of the present application, the operation of the second model needs to be completed in the video memory. When the second model is deployed in the running memory, the model training device cannot run the second model. In one possible implementation, the model training device divides the second model into m fifth sub-models using PP segmentation. The m fifth sub-models are respectively stored in the running memory of m devices, and the fifth sub-models stored in any device are different. Specifically, the model training device segments the tensors in the second model along one dimension. For example, when the tensor of the second model is a weight matrix, the weight matrix is ​​segmented along the row direction. For another example, when the tensor of the second model is a weight matrix, the weight matrix is ​​segmented along the column direction. In another possible implementation, the model training device deploys the second model in the running memory of any device among the m devices.

[0168] 203. Deploy the first model in the video memory of n devices.

[0169] In the embodiment of the present application, the first model is deployed in the video memory of n devices. When the first model is deployed in the video memory, the model training device can run the first model, that is, the model training device can use the first model for reasoning or training.

[0170] In the embodiment of the present application, the training object of the model training device is the first model, and before training the first model, it is necessary to first obtain the inference result by executing the inference process of the first model, and then determine the loss required for training the first model based on the inference result. After obtaining the loss, the training of the first model is completed by updating the parameters of the first model based on the loss, wherein the inference process of the first model refers to using the first model to perform a natural language processing task. Therefore, the model training device deploys the first model in the video memory of n devices in order to execute the inference process of the first model.

[0171] It should be understood that the first model deployed in the running memory of i devices and the first model deployed in the video memory of n devices are two identical models, that is, when the first model is deployed in the running memory of i devices, the first model is also deployed in the video memory of n devices.

[0172] In the embodiment of the present application, the n devices deploying the first model include the i devices deploying the first model in step 201 and the m devices deploying the second model in step 202, where n is the sum of m and i, and the i device is different from any two devices in the m devices. Figure 3 , Figure 3 Another distributed training system architecture diagram provided in the embodiment of the present application. Figure 3 As shown, the distributed training system includes 3 devices, among which the running memory of 2 devices deploys the first model, the running memory of 1 device deploys the second model, and the video memory of the 3 devices deploys the first model. Specifically, the video memory of device 1, the video memory of device 2, and the video memory of device 3 all deploy the first model, the running memory of device 1 and the running memory of device 2 all deploy the first model, and the running memory of device 3 deploys the second model. At this time, n=3, i=2, and m=1.

[0173] In one possible implementation, the model training device uses a tensor parallel (TP) splitting method to split the first model to obtain n first sub-models. The n first sub-models are stored in the video memory of n devices respectively, and the first sub-models in the video memory of any two devices are different. Specifically, the model training device splits the tensor in the first model along one dimension. For example, when the tensor in the first model is a weight matrix, the model training device splits the weight matrix along the row direction to obtain n first sub-models. For another example, when the tensor in the first model is a weight matrix, the model training device splits the weight matrix along the column direction to obtain n first sub-models.

[0174] 204. Using the first model deployed in the video memory of the n devices, process the training prompt words to generate a first text.

[0175] In the embodiment of the present application, the training prompt word is a prompt word used to guide the first model to perform a natural language task. For example, the training prompt word is a prompt word used to guide the first model to perform text classification. For another example, the training prompt word is a prompt word used to guide the first model to perform question answering. For another example, the training prompt word is a prompt word used to guide the first model to generate a text summary.

[0176] When the training prompt words are determined, the training prompt words are used as inputs of the first model in the video memory of the n devices, and the first model is used to execute the natural language task in the training prompt words to generate the first text, thereby completing the reasoning process of the first model, wherein the first text is the result obtained after the first model executes the natural language task in the training prompt words, that is, the reasoning result described in step 203.

[0177] Optionally, when the first model is divided into n first sub-models and the n first sub-models are deployed in the video memory of n devices, the training prompt words are processed using the n first sub-models deployed in the video memory of the n devices to generate the first text. In a possible implementation, the n devices include a shared device, the video memory of the shared device includes a shared storage space, the n devices can access the shared storage space, and the n first sub-models are stored in the shared storage space. For each of the n devices, the target data required to complete the target processing of the corresponding first sub-model is obtained from the shared storage space, wherein the first sub-model corresponding to the device is the first sub-model deployed in the video memory of the device, and the target processing is used to realize the processing of the training prompt words by the first model. Specifically, the target processing is the processing performed by each first sub-model in the process of the first model processing the training prompt words to generate the first text. For example, the n devices include device 1 and device 2, and the n first sub-models include first sub-model a and first sub-model b, wherein the first sub-model a is deployed in the video memory of device 1, and the first sub-model b is deployed in the video memory of device 2. At this time, the first sub-model corresponding to device 1 is the first sub-model a, and the first sub-model corresponding to device 2 is the first sub-model b. In the process of the first model processing the training prompt word to generate the first text, the first sub-model a needs to process the training prompt word to obtain the intermediate processing result, and the first sub-model b needs to process the intermediate processing result to generate the first text. Then, for the second first model a, the target processing is to process the training prompt word, and for the second first model b, the target processing is to process the intermediate processing result. For device 1, the target data to be obtained is the training prompt word, and for device 2, the target data to be obtained is the intermediate processing result. Optionally, for each of the n devices, the data required to complete the target processing of the corresponding first sub-model is determined from the shared storage space as the target data. For each of the n devices, the target data stored in the shared storage space is stored in the video memory of the device, so that the target data can be processed in the video memory later. After each device obtains the required target data from the shared storage space, the n copies of the first sub-model deployed in the video memory of the n devices are used to perform target processing on the corresponding target data to generate the first text.

[0178] Optionally, n copies of the first sub-model are stored in the shared storage space. In the case where n copies of the first sub-model are stored in the shared storage space, the model training device realizes allgathering of data generated by executing target processing on the n copies of the first sub-model, and stores the results generated by executing the target processing on the n copies of the first sub-model in the shared storage space. Each device can then obtain the required target data from the shared storage space.

[0179] Optionally, the graphics card of each device can communicate with the graphics card of the shared device. For the convenience of expression, the communication between graphics cards will be referred to as graphics card communication below. Each device obtains the required target data from the shared storage space through the graphics card communication with the shared device, and stores the target data in the video memory of each device. Optionally, graphics card communication is implemented through the NVIDIA Collective Communications Library (NCCL). The target data in the shared storage space is stored in the video memory of each device through graphics card communication, and there is no need to use the read resources of the input / output (I / O) interface of each device to read the target data from the shared storage space, where the read resources are resources used to read data. This can reduce the pressure on the read resources of the I / O interface of n devices and achieve load balancing of n devices.

[0180] Optionally, the model training device performs step 204 under the framework of a versatile Large Language Model (vLLM). Specifically, the model training device uses the first model deployed in the video memory of n devices under the vLLM framework to process the training prompt words and generate the first text. Since the first model is a language model, and the language model includes an attention mechanism, the process of using the first model to process the training prompt words involves the processing of the data by the attention mechanism, and specifically, it involves the processing of the data by the key and value in the attention mechanism. When the language model is run under the vLLM framework, the continuous keys and continuous values ​​in the attention mechanism can be stored in the discontinuous video memory space through the PagedAttention technology, and then the keys and values ​​in the attention mechanism can be stored in the video memory of n devices respectively, so that the processing of the data by the attention mechanism can be completed in parallel by n devices. In this way, the processing speed of the attention mechanism on the data can be improved, and the speed of the first model processing the training prompt words to generate the first text can be improved.

[0181] 205. Transfer the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices.

[0182] In an embodiment of the present application, the model training device generates a first text using the first model in the video memory of n devices, indicating that the first model in the video memory of n devices has completed a reasoning process, and then the loss required for training the first model is determined based on the reasoning result (i.e., the first text), and the first model is trained based on the loss. Since the loss required for training the first model needs to be obtained by running the second model, and the operation of the second model and the training of the first model based on the loss need to be completed in the video memory of the device, the model training device transfers the first model deployed in the video memory of n devices from the video memory of n devices to the running memory of n devices after generating the first text, thereby releasing the video memory in n devices for running critics and training the first model. Optionally, when the first model is divided into n first sub-models and the n first sub-models are deployed in the video memory of n devices, the first sub-model in each device is transferred from the video memory to the running memory.

[0183] 206. Transfer the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices.

[0184] In an embodiment of the present application, after the model training device generates the first text using the first model in n devices, the first model in i devices is transferred from the running memory in the i devices to the video memory, so as to subsequently train the first model, that is, update the parameters of the first model.

[0185] Optionally, when the first model is divided into i second sub-models and the i second sub-models are deployed in the running memory of i devices, the second sub-models in the i devices are transferred from the running memory to the video memory.

[0186] 207. Transfer the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices.

[0187] In an embodiment of the present application, after the model training device generates the first text using the first model in n devices, the second model in m devices is transferred from the running memory in the m devices to the video memory, so as to obtain the loss required for training the first model by running the second model.

[0188] Optionally, when the second model is divided into m fifth sub-models and the m fifth sub-models are deployed in the running memory of m devices, the fifth sub-models in the m devices are transferred from the running memory to the video memory.

[0189] 208. Using the second model in the video memory of the m devices, process the training prompt words and the first text, and estimate a first improvement degree of the target word in the first text to the target matching degree.

[0190] In the embodiment of the present application, the target word represents any word in the first text. After the training prompt word and the first text are input into the second model, the second model can estimate the improvement degree of the target word in the first text to the target matching degree, that is, the first improvement degree, wherein the higher the improvement degree of the target word to the target matching degree, the higher the first improvement degree.

[0191] 209. The i devices determine a first loss of the first model based on the first improvement.

[0192] The higher the first lift, the higher the lift of the target word generated by the first model to the target match, which means the target word generated by the first model has a higher accuracy. Therefore, the i devices can determine the loss of the first model based on the first lift, which is the first loss. Optionally, the first loss is negatively correlated with the first lift. Optionally, the model training device controls the i devices to execute step 209.

[0193] Optionally, the second model may estimate the improvement of the target matching degree of each word in the generated first text, and then determine the first loss based on all the improvements. In one possible implementation, all the improvements are summed to obtain the total improvement, and the first loss is determined based on the total improvement, wherein the first loss is negatively correlated with the total improvement. In another possible implementation, all the improvements are averaged to obtain the average improvement, and the first loss is determined based on the average improvement, wherein the first loss is negatively correlated with the average improvement.

[0194] 210. The i devices update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model.

[0195] In an embodiment of the present application, the target model is an LLM obtained by training the first model. Specifically, the target model is an LLM obtained by updating the parameters of the first model based on the first loss. Since the target model is obtained by updating the parameters of the first model based on the first loss, the text generated by the target model has a high degree of matching with the preset conditions. In one possible implementation, i devices update the parameters of the first model to reduce the first loss until the first loss converges to obtain the target model. In another possible implementation, i devices determine the gradient of the first model based on the first loss. Based on the gradient of the first model, reverse gradient propagation is performed on the first model, and the parameters of the first model are updated until the loss of the first model converges to obtain the target model. Optionally, the model training device controls i devices to execute step 210.

[0196] In an embodiment of the present application, the model training device deploys the first model in the running memory of i devices, the second model in the running memory of m devices, and the first model in the video memory of n devices, wherein the n devices include i devices and m devices. After the first model deployed in the video memory of n devices is used to process the training prompt word to generate the first text, the reasoning process of the first model is completed. Then the first model deployed in the video memory of n devices is transferred from the video memory of n devices to the running memory of n devices, thereby releasing the video memory in n devices. After the video memory in n devices is released, the first model deployed in the running memory of i devices is transferred from the running memory of i devices to the video memory of i devices, and the second model deployed in the running memory of m devices is transferred from the running memory of m devices to the video memory of m devices. Then, the second model in the video memory of m devices is used to process the training prompt word and the first text, and the first improvement degree of the target word in the first text to the target matching degree is estimated. The first loss of the first model is determined based on the first improvement degree, and the first improvement degree is negatively correlated with the first loss. Based on the first loss, the parameters of the first model in the video memory of i devices are updated to obtain the target model, thereby completing the training of the first model. Based on the technical solution provided in the embodiment of the present application, the reasoning of the first model and the training of the first model can be completed by n devices, and because the reasoning process of the first model and the training process of the first model are not performed simultaneously, the technical solution can alternately perform the reasoning process and the training process through n devices, thereby improving the utilization rate of device resources, thereby reducing the device resources required to complete the reasoning of the first model and the training of the first model.

[0197] The traditional method usually completes the target training process by running the second model through i devices, executing the training process of the first model through m devices, and executing the reasoning process of the first model through N_vllm devices. Obviously, the technical solution provided by the embodiment of the present application is less than the traditional method in the number of devices required to complete the target training process, so on the one hand, the technical solution provided by the embodiment of the present application can reduce costs compared to the traditional method. On the other hand, in the case where the number of devices that can be used to complete the target training process is n, the technical solution provided by the embodiment of the present application can execute the reasoning process of the first model through n devices, run the second model through i devices, and execute the training process of the first model through m devices, but the traditional method is to execute the reasoning process of the first model, run the second model, and execute the training process of the first model through different devices, so the traditional method needs to divide n devices into three parts, wherein one part is used to execute the reasoning process of the first model, another part is used to run the second model, and another part is used to execute the training process of the first model. Obviously, compared with the traditional method, the technical solution provided in the embodiment of the present application uses a larger number of devices to execute the reasoning process of the first model, so the time spent on completing the reasoning process of the first model is less, which can increase the speed of the reasoning process of the first model, thereby increasing the speed of the target training process.

[0198] As an optional implementation, when i copies of the second sub-model deployed in i devices are transferred from the running memory in the i devices to the video memory, the model training apparatus performs the following steps:

[0199] 301. Transfer the first training information stored in the running memory of the i devices to the video memory of the i devices.

[0200] In an embodiment of the present application, the first training information includes: parameters of the second sub-model, the gradient of the first model in the previous round of training, and the current number of training rounds. In this embodiment, the running memory of the i devices not only stores i copies of the parameters of the second sub-model, but also stores the gradient of the first model in the previous round of training and the current number of training rounds. It should be understood that the target training process usually includes multiple rounds of training, and each round of training needs to be trained on the basis of the previous round. Therefore, the model training device needs to store the first training information in the video memory of the i devices, so as to subsequently train the first model in the video memory of the i devices.

[0201] The gradient of the first model in the previous round of training represents the gradient of the first model determined based on the first loss in the previous round of training before the current training round. The model training device can update the first model of the current training round based on the gradient in the previous round of training and the gradient of the current training round.

[0202] The current number of training rounds can be used to determine the learning rate of the current number of training rounds of the first model, wherein the learning rate is a parameter that affects the convergence speed of the first model. Specifically, the more rounds that have been trained, the smaller the learning rate of the current number of training rounds, and correspondingly, the slower the convergence speed of the first model. In the case where the first training information includes the current number of training rounds, the model training device can determine the learning rate of the current number of training rounds of the first model based on the current number of training rounds, and the number of rounds can then be based on the learning rate of the current number of training rounds of the first model, and the first model can be trained for the current number of rounds.

[0203] As an optional implementation, the above i copies of the third sub-model are obtained through training of the current number of training rounds. In the case where i copies of the second sub-model are deployed in the video memory of n devices, the model training device performs the following steps:

[0204] 401. When the current training round number is the last training round number, merge the i third sub-models to obtain the target model.

[0205] In this embodiment, the current training round number is the last training round number, indicating that the target training process can be completed by completing the current training round number to obtain the target model. Therefore, in the case where i third sub-models are obtained through the training of the current training round number, the model training device uses the model obtained by merging the i third sub-models as the target model. Specifically, the i first sub-models are obtained by segmenting the first model. Therefore, after obtaining the i third sub-models by updating the parameters of the i first sub-models, the i third sub-models are merged in accordance with the segmentation method of the i second first models to obtain the target model. For example, the i first sub-models are obtained by segmenting the first model in accordance with the PP segmentation method. Therefore, after obtaining the i third sub-models by updating the parameters of the i first sub-models, the i third sub-models are merged in accordance with the PP segmentation method to obtain the target model.

[0206] As an optional implementation, when n copies of the second sub-model are deployed in the video memory of n devices, the model training apparatus further performs the following steps:

[0207] 501. When the current training round number is not the last training round number, transfer the i copies of the third sub-model from the video memory of the i devices to the running memory of the i devices.

[0208] In the embodiment of the present application, the current training round is not the last training round, which means that after completing the current training round, training needs to be continued to complete the training of the target training process to obtain the target model. Therefore, when i copies of the third sub-model are obtained through the training of the current training round, the model training device transfers i copies of the third sub-model from the video memory of i devices to the running memory of i devices, and releases the video memory of i devices for executing the reasoning process of the first model in the next round of training.

[0209] 502. Each of the n devices executes the step of merging the i copies of the third sub-model in the running memory of the i devices, so that the running memory of each of the n devices stores the updated first model.

[0210] In the embodiment of the present application, the updated first model is the first model after the parameters are updated through the training of the current training round. Each of the n devices performs the step of merging i copies of the third sub-model to obtain the updated first model, and stores the updated first model in the running memory, so that the running memory of each device stores the updated first model.

[0211] 503. Each of the n devices executes the step of dividing the updated first model into n fourth sub-models, so that a copy of the fourth sub-model is stored in the running memory of each of the n devices, and the fourth sub-models stored in the running memory of different devices among the n devices are different.

[0212] In an embodiment of the present application, when the updated first model exists in the running memory of each of the n devices, the model training device enables each device to perform a segmentation operation on the updated first model. Specifically, the updated first model in each device is segmented into n fourth sub-models. In a possible implementation, the model training device segments the updated first model according to the segmentation method of TP to obtain n fourth sub-models. After being segmented into n fourth sub-models, a fourth sub-model is stored in the running memory of each device, and the fourth sub-models stored in any two devices are different. For example, the running memory of device 1 and device 2 both store the updated first model. After device 1 and device 2 both segment the updated first model into the fourth sub-model x1 and the fourth sub-model x2, the fourth sub-model x1 is stored in the running memory of device 1, and the fourth sub-model x2 is stored in the running memory of device 2.

[0213] 504. Each of the n devices executes to store the fourth sub-model stored in the running memory into the shared storage space.

[0214] In an embodiment of the present application, when a copy of the fourth sub-model is stored in the running memory of each of the n devices, each device stores the fourth sub-model stored in the running memory to the shared storage space, so that the complete first model can be stored in the shared storage space, so that each device can use the parameters of the first model in the shared storage space for the next round of training.

[0215] Since each of the n devices stores a copy of the fourth sub-model in the shared storage space, compared to the situation where all n devices store the updated first model stored in the running memory in the shared storage space, this can reduce the amount of data that each device needs to store in the shared space of the shared device, thereby reducing the pressure on the write resources of the I / O interface of the shared device, where the write resources are resources used to write data, thereby achieving load balancing of the n devices.

[0216] As an optional implementation, the model training device performs the following steps in the process of transferring m copies of the fifth sub-model from the running memory of n devices to the video memory of m devices:

[0217] 601. Transfer the second training information stored in the running memory of the m devices to the video memory of the m devices.

[0218] In an embodiment of the present application, the second training information includes: the parameters of the fifth sub-model (checkpoint), the gradient of the second model in the previous round of training (optimizer), the current number of training rounds (scheduler), and the first training data (dataloader) that the fifth sub-model needs to process in the current number of training rounds, wherein the first training data is obtained by segmenting the training prompt words and the first text, and the segmentation method of the training prompt words and the first text is the same as the segmentation method of the second model. Since the target training process usually includes multiple rounds of training, each round of training needs to be trained on the basis of the previous round, so the model training device needs to store the second training information in the video memory of m devices, so as to subsequently train the second model in the video memory of the m devices, thereby enabling the target training process to be executed uninterruptedly.

[0219] The gradient of the second model in the previous round of training represents the gradient of the second model determined based on the second loss of the previous training round before the current round. The model training device can update the second model of the current round according to the gradient in the previous round of training and the gradient of the current training round.

[0220] The current training round number can be used to determine the learning rate of the current training round number of the second model, wherein the learning rate is a parameter that affects the convergence speed of the second model. In the case where the second training information includes the current training round number, the model training device can determine the learning rate of the current training round number of the second model according to the current training round number, and then can train the second model for the current round number based on the learning rate of the current training round number of the second model.

[0221] As an optional implementation, a third model (reward model) is deployed in the video memory of the target device among the n devices, wherein the third model is used to evaluate the degree of match between the text generated by the first model and the preset requirements. Optionally, the third model outputs a reward value to characterize the degree of match between the text and the preset requirements, wherein the reward value is positively correlated with the degree of match. For example, after the training prompt words and the text generated by the first model are input into the third model, the third model outputs a reward value, which characterizes the degree of match between the text generated by the first model and the preset requirements. Optionally, the third model may also be stored outside the n devices, and the model training device may call the third model through an application programming interface (API) and run the third model.

[0222] After transferring m copies of the fifth sub-model from the running memory of n devices to the video memory of m devices, the model training apparatus performs the following steps:

[0223] 701. Obtain a second improvement degree.

[0224] In the embodiment of the present application, the second lift is the lift of the target word to the target match, and the second lift is obtained by processing the training prompt word and the first text by the third model. Optionally, the training prompt word and the first text are input into the third model so that the third model determines the second lift of the target word in the first text to the target match.

[0225] 702. Determine a second loss of the second model based on a first difference between the second lift and the first lift.

[0226] The greater the difference between the first lift and the second lift, the lower the accuracy of the result output by the second model. Therefore, the model training device can determine the second loss of the second model according to the first difference between the second lift and the first lift, wherein the second loss is positively correlated with the first difference.

[0227] 703. Based on the second loss, update the parameters of the m copies of the fifth sub-model in the video memory of the m devices to obtain m copies of the sixth sub-model.

[0228] Since the second loss is obtained based on the first difference between the second lift and the first lift, and the first difference can characterize the accuracy of the result output by the second model, the model training device updates the parameters of the second model based on the second loss, which can improve the accuracy of the result output by the second model. Therefore, the model training device updates the sub-model of the second model based on the second loss. Specifically, by updating the parameters of the m parts of the fifth sub-model in the video memory of m devices, the m parts of the sixth sub-model are obtained. In one possible implementation, the model training device makes the second loss smaller by updating the parameters of the m parts of the fifth sub-model until the second loss converges to obtain the m parts of the sixth sub-model. In another possible implementation, the model training device determines the gradient of the m parts of the fifth sub-model based on the second loss. Based on the gradient of the m parts of the fifth sub-model, the m parts of the fifth sub-model are reversely propagated, and the parameters of the m parts of the fifth sub-model are updated until the loss of the m parts of the fifth sub-model converges to obtain the m parts of the sixth sub-model.

[0229] 704. Merge the m sixth sub-models to obtain an updated second model.

[0230] In an embodiment of the present application, when m portions of the sixth sub-model are obtained through training, the m portions of the sixth sub-model are merged to obtain an updated second model, that is, the second model after the parameters are updated. Specifically, the m portions of the fifth sub-model are obtained by segmenting the second model, so after obtaining the m portions of the sixth sub-model, the m portions of the sixth sub-model are merged in the manner of segmenting to obtain the m portions of the fifth sub-model, to obtain the updated second model. For example, the m portions of the fifth sub-model are obtained by segmenting the second model in the PP segmentation manner, so after obtaining the m portions of the sixth sub-model, the m portions of the sixth sub-model are merged in the PP segmentation manner to obtain the updated second model.

[0231] In this implementation, after obtaining the second lift output by the third model, the model training device determines the second loss of the second model based on the first difference between the second lift and the first lift. Based on the second loss, the parameters of the m copies of the fifth sub-model in the video memory of the m devices are updated to obtain the m copies of the sixth sub-model, and the updated second model is obtained by merging the m copies of the sixth sub-model, so as to improve the accuracy of the second model in estimating the target matching degree.

[0232] As an optional implementation, the model training device performs the following steps before executing step 305:

[0233] 801. Deploy a third model in the running memory of a target device among the n devices.

[0234] In the embodiment of the present application, the target device is any one of the n devices, that is, the third model can be deployed in the running memory of any one of the n devices.

[0235] The model training device further performs the following steps after executing step 305 and before executing step 30:

[0236] 802. Transfer the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device.

[0237] After the model training device transfers the first model deployed in the video memory of n devices from the video memory of n devices to the running memory of n devices by executing step 305, the video memory of n devices can be released, including releasing the video memory of the target device. The model training device then transfers the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device, so as to subsequently run the third model.

[0238] 803. Process the training prompt words and the first text using the third model deployed in the video memory of the target device to obtain a second lift.

[0239] The second lift degree and the third model in this step are the same as the second lift degree and the third model in step 701, and will not be described again here.

[0240] 804. Using the second model in the video memory of the m devices, process the training prompt words and the first text to estimate a third improvement degree of the reference words in the first text to the target matching degree.

[0241] In the embodiment of the present application, the reference word is the word preceding the target word. For example, the first text is: Cheetah is a cat family animal, if the reference word is cat, then the target word is family.

[0242] After obtaining the second lift and the third lift, the model training device performs the following steps during the execution of step 309:

[0243] 805. Determine an advantage function based on a second difference between the second lift and the third lift.

[0244] In an embodiment of the present application, the advantage function represents the accuracy of generating a target word in the next word of the reference word when a reference word has been generated. For example, the first text is: Cheetah is a cat family animal. If the reference word is cat, then the target word is family. Since the first model generates the first text by generating one word after another in the process of generating the first text, the first model first generates cat and then family. At this time, the advantage function represents the improvement of the first model based on the generation of the word family to the target match when the word cat has been generated. Because the third improvement represents the improvement of the target match when the reference word is generated based on the reference word to generate a complete first text, the second improvement represents the improvement of the target match when the target word is generated, so the difference between the second improvement and the third improvement represents the improvement of the target match when the target word is generated based on the reference word. Therefore, the model training device can determine the advantage function based on the second difference between the second improvement and the third improvement. Optionally, the advantage function is positively correlated with the second difference.

[0245] In one possible implementation, the model training device obtains an action value function of the target word based on the second lift, obtains a state value function of the reference word based on the third lift, and determines the difference between the action value function and the state value function to obtain an advantage function.

[0246] 806. Determine the first loss based on the advantage function.

[0247] Because the first model generates text one word at a time, how to determine what words to generate next based on the words that have been generated has a great impact on the target matching of the text that is finally generated. The advantage function represents the improvement of the target matching degree by the first model generating the target word based on the reference word. Therefore, based on the advantage function, the first loss is determined, and then the parameters of the first model are updated based on the first loss, which can improve the target matching degree of the text generated by the first model. Optionally, the first loss is negatively correlated with the advantage function.

[0248] As an optional implementation, before the target training process is executed, the first model is an SFT model. The third model outputs a reward value. Since the reward value output by the third model for a long and meaningless text has an error, that is, the reward value output by the third model represents a high degree of match between the long and meaningless text and the preset requirements. For example, the training prompt word is "Please help me write a review of the goods", and the first text generated by the first model based on the training prompt word is "Fragile goods are of the quality of the goods". For the training prompt word, the first text is a long and meaningless text, but the reward value output by the third model represents a high degree of match between the first text and the preset requirements. Since the second lift is the result of the output of the third model, when the first loss is obtained based on the first difference between the second lift and the first lift, if there is an error in the result output by the third model, then there is an error in the first loss. Further, when the parameters of the first model are updated based on the first model, it is easy to cause the result output by the first model to include long and meaningless text.

[0249] In order to reduce the influence of the error of the result output by the third model on the result output by the first model, the model training device uses the SFT model as a reference model to constrain the result output by the first model. Specifically, the model training device can obtain the second lifting degree by the following steps: determining the Kullback-Leibler (KL) divergence between the result output by the first model and the result output by the SFT model. Optionally, the training prompt word and the first text are input to the first model so that the first model outputs a first probability sequence, wherein the first probability sequence is the confidence of each word in the first text, wherein the first model determines the probability of each word in the first text appearing in the first text, and calculates the logarithmic function of the probability as the confidence, for example, p represents the probability of a word in the first text appearing in the first text, then the confidence of the word in the first text is logp, wherein log represents the logarithmic function. The training prompt word and the first text are input to the SFT model so that the SFT model outputs a second probability sequence, wherein the second probability sequence is the confidence of each word in the first text. Optionally, the SFT model determines the probability that each word in the first text appears in the first text, and calculates the logarithmic function of the probability as the confidence. The second lift is then obtained by weighted summing the KL divergence and the reward value output by the third model. At this time, based on the first difference between the second lift and the first lift, the first loss is obtained, and the parameters of the first model are updated based on the first loss. The KL divergence can be used to make the difference between the result output by the first model and the result output by the SFT model smaller. Since the SFT model is obtained through supervised fine-tuning, the result output by the SFT model is reasonable and logical. Therefore, by making the difference between the result output by the first model and the result output by the SFT model smaller, the result output by the first model can be made reasonable and logical. This can reduce the impact of the error of the result output by the third model on the result output by the first model.

[0250] Optionally, before the first model deployed in the video memory of n devices is transferred from the video memory of n devices to the running memory of n devices, the SFT model can be stored in the running memory of any one of the n devices. After the first model deployed in the video memory of n devices is transferred from the video memory of n devices to the running memory of n devices, the SFT model can be transferred from the running memory to the video memory, and then the SFT model is run to obtain the SFT output result.

[0251] Optionally, the SFT model may also be stored outside the n devices, and the model training device may call the SFT model through an API, and obtain the SFT output result by running the SFT model.

[0252] See also Figure 4 , Figure 4A flowchart of a model training provided in an embodiment of the present application. Figure 4 The process shown is the process of the target training process, such as Figure 4 As shown, the target training process includes the following four stages: 1. Experience acquisition stage

[0253] (make-experience), 2. training phase (training), 3. parameter saving phase (param_save), 4. parameter loading phase (param_load), wherein make-experience includes the reasoning process of the first model, training includes the training process of the first model and the operation of the second model, param_save includes saving the parameters of the first model updated by training and the parameters of the updated second model, and param_load includes loading the parameters of the first model saved in the running memory into the video memory to execute the next round of the reasoning process of the first model. Specifically, the model training device deploys the first model in the video memory of n devices as Figure 4 The first model is used for inference in i and deployed in the running memory of i devices as Figure 4 The first model for training in the m devices is deployed in the running memory of the m devices as the second model for training. Figure 4As shown, i=m=n / 2. In make-experience, the model training device uses the first model for inference deployed in the video memory of n devices to process the training prompt words sampled from the training data set to generate a first text, thereby completing make-experience. After completing make-experience, the model training device performs a data exchange (offloading) operation. Specifically, the first model for inference in the video memory of n devices is transferred to the running memory of n devices, and then the first model for training in n / 2 devices is transferred from the running memory to the video memory, and the second model for training in n / 2 devices is transferred from the running memory to the video memory. After performing the data exchange operation, the model training device performs training. Specifically, while transferring the first model for training from the running memory to the video memory, the first training information is also stored in the video memory, wherein the first training information includes: the parameters of the first model, the gradient of the first model in the previous round of training, and the current number of training rounds. Similarly, while transferring the second model for training from the running memory to the video memory, the second training information will be stored in the video memory, wherein the second training information includes: the parameters of the second model for training, the gradient of the second model in the previous round of training, the current training round number, and the first training data of the second model in the above-mentioned current training round number. In this way, the training information of the first model and the training information of the second model can be restored (resume) to the latest state in the current training round number, thereby enabling the target training process to be executed uninterruptedly. After transferring the first model for training to the video memory and storing the first training information in the video memory, the model training device calculates the first loss according to steps 208 and 209, and updates the parameters of the first model for training according to the first loss. After transferring the second model for training to the video memory and storing the second training information in the video memory, the model training device determines the second loss according to steps 701 to 703, and updates the parameters of the second model for training according to the second loss. The training phase is completed after the parameters of the first model and the parameters of the second model are updated. After completing the training phase, the data exchange operation is performed again. Specifically, the first training model in the video memory of the n / 2 devices is transferred to the running memory, and the second training model in the video memory of the n / 2 devices is transferred to the running memory. After performing the data exchange operation, the model training device executes param_save. After the first model in the video memory of the n / 2 devices is transferred to the running memory, a merge operation is performed. Specifically, since the first training model in the n / 2 devices is obtained by splitting the first model according to the PP splitting method, each of the n devices merges the first models in the running memory of the n / 2 devices according to the PP splitting method, and the first model after the parameters are updated can be obtained.In this way, the running memory of any one of the n devices stores the first model after the parameters are updated. Optionally, the specific implementation steps of the merging operation can refer to step 501 and step 502. Each of the n devices then performs a splitting operation. Specifically, each of the n devices splits the first model stored in the running memory according to the TP splitting method to obtain n first sub-models, and each of the n devices stores one of the n fourth sub-models in the running memory. The fourth sub-models stored in the running memory of any two devices are different. Optionally, the splitting process can refer to step 503. Each of the n devices stores the fourth sub-model stored in the running memory to the shared storage space of the shared device, so that in the subsequent process of performing the next round of make-experience through the n fourth sub-models, the n devices communicate with the graphics card between the shared devices to obtain the target data from the shared storage space. Optionally, the specific implementation process of this step can refer to step 504. After each of the n devices stores the fourth sub-model stored in the running memory to the shared storage space, param_save is completed. After completing param_save, the model training device performs a data exchange operation, specifically, transferring the n copies of the fourth sub-model stored in the running memory of the n devices to the video memory. After performing the data exchange operation, the model training device executes param_load. Specifically, the parameters of the fourth sub-model in the video memory of the n devices are updated to the parameters of the fourth sub-model in the shared storage space through the allgather operation. In this way, when each of the n devices can access the shared storage space, each of the n devices can obtain the corresponding fourth sub-model from the shared storage space in the next round of training, and deploy the obtained fourth sub-model in the video memory, so that the n devices can use the fourth sub-model deployed in the video memory to perform the inference process of the next round of training.

[0254] Figure 4 The target training process shown is implemented based on a distributed training framework. Since the first model and the second model need to be split during the target training process, a distributed training framework that supports splitting can be selected. Figure 4The distributed training framework used in the target training process shown is the nemo-megatron framework. After the first model and the second model are segmented, different sub-models need to be deployed on different devices, and the training of the first model and the operation of the second model need to be implemented asynchronously, so it can be implemented on the basis of the distributed training framework in combination with the support of the distributed computing framework. Optionally, the distributed computing framework is a ray framework, and the function calls and / or class calls involved in the operation of the second model can be accelerated through the ray framework. Optionally, in order to increase the speed at which the first model processes the training prompt words to generate the first text and speed up the make-experience, make-experience can be completed under the vLLM framework. After the deployment of the distributed training system is completed based on the distributed training framework, the model training device starts the distributed training through a launcher. Optionally, the launcher is an accelerate launcher.

[0255] Optionally, the target training process involves the first model, the second model, the SFT model, and the third model. The distributed training framework is the nemo-megatron framework, the distributed computing framework is the ray framework, and the make-experience is completed under the vLLM framework. Then the make-experience stage of the first model is completed under the vLLM framework, the training process of the first model is completed under the nemo-megatron framework, the operation of the second model is completed under the ray framework, the operation of the SFT model is completed under the nemo-megatron framework, and the operation of the third model is completed under the vLLM framework.

[0256] In a possible implementation scenario, the target training process is as follows Figure 4 As shown, it includes the following four stages: make-experience, training, param_save, param_load. Correspondingly, the time consumption of the target training process is the sum of the time consumption of make-experience, the time consumption of training, the time consumption of param_save, and the time consumption of param_load. Considering that the time consumption of param_save and param_load are both in the second level, and the time consumption of make-experience and training are both in the minute level, that is, the time consumption of param_save and param_load are shorter than the time consumption of make-experience and training, the time consumption of the target training process can be approximately equal to the sum of the time consumption of make-experience and training.

[0257] If the time taken to execute the inference process of the first model using one device is V_vllm minutes, the time taken to execute the training process of the first model using n devices and the time taken to run the second model using n devices are both V_train minutes.

[0258] In the technical solution provided in the embodiment of the present application, n devices can be used to execute the reasoning process of the first model in parallel, so the time consumption of make-experience is V_vllm / n minutes. If the minimum capacity for running the first model and the minimum capacity for running the second model are both n / 2, then i=m=n / 2, and accordingly, the time consumption for executing the training process of the first model in parallel using n / 2 devices and the time consumption for running the second model in parallel using n / 2 devices are both 2V_train minutes. Since the training process of the first model and the operation of the second model can be executed in parallel, the time consumption of training is 2V_train. At this time, the time consumption of the target training process is the sum of the time consumption of make-experience and the time consumption of training: (V_vllm / n+2V_train) minutes, and the average time consumption of each device is: (V_vllm / n+2V_train) / n minutes. In the technical solution provided in the embodiment of the present application, the description of each stage in the target training process and the time consumption of each stage are shown in Table 1 below.

[0259]

[0260] Table 1

[0261] In the traditional method, if N_vllm devices are used to execute the inference process of the first model in parallel, n / 2 devices are used to execute the training process of the first model in parallel, and n / 2 devices are used to run the second model in parallel. Then the time consumption of make-experience is V_vllm / N_vllm, and the time consumption of training is 2V_train. At this time, the time consumption of the target training process is the sum of the time consumption of make-experience and the time consumption of training: (V_vllm / N_vllm+2V_train) minutes, and the average time consumption of each device is: (V_vllm / N_vllm+2V_train) / (n+N_vllm) minutes.

[0262] By comparison, it can be seen that the technical solution provided in the embodiment of the present application consumes n devices, while the device resources consumed by the traditional method are (n+ / N_vllm) devices, that is, the technical solution provided in the embodiment of the present application consumes fewer device resources than the traditional method.

[0263] If the equipment resources consumed by the technical solution provided in the embodiment of the present application and the equipment resources consumed by the traditional method are both n devices, the traditional method uses N_vllm devices to execute the reasoning process of the first model in parallel, uses (n-N_vllm) / 2 devices to execute the training process of the first model in parallel, and uses (n-N_vllm) / 2 devices to run the second model in parallel. Then the time consumption of make-experience of the traditional method is V_vllm / N_vllm, and the time consumption of training of the traditional method is: At this point, the time consumption of the target training process of the traditional method is the sum of the time consumption of make-experience and the time consumption of training: The average time consumed by each device using the traditional method is: If custom_cost_time_per_gpu is used to represent the average time consumption of each device of the technical solution provided in the embodiment of the present application, openrlhf_cost_time_per_gpu is used to represent the average time consumption of each device of the traditional method, and speed_up is used to represent the relative improvement of the average time consumption of each device of the technical solution provided in the embodiment of the present application compared with the average time consumption of each device of the traditional method, then speed_up satisfies the following formula: speed_up = (openrlhf_cost_time_per_gpu-custom_cost_time_per_gpu) / openrlhf_cost_time_per_gpu. custom_cost_time_per_gpu=

[0264] (V_vllm / n+2V_train) / n minutes, and / n is greater than (V_vllm / n+2V_train) / n, so speed_up is greater than 0, which shows that the technical solution provided by the embodiment of the present application can reduce the time consumption of each device and improve the speed of each device on the basis of the traditional method. Optionally, when n=32 and N_vllm=8, / 32=(V_vllm / 8+8V_train / 3) / 32,custom_cost_time_per_gpu=(V_vllm / 32+2V_train) / 32。openrlhf_cost_time_per_gpu-custom_cost_time_per_gpu=(3V_vllm / 32+2V_train / 3) / 32,speed_up=(3V_vllm / 32+2V_train / 3) / (V_vllm / 8+8V_train / 3). If batchsize is used to represent the number of training prompt words input into the first model during the inference process of the first model, and rollout_batchsize is used to represent the number of training prompt words sampled from the training data set, then when batchsize=rollout_batchsize, the time consumption of the target training process of the technical solution provided in the embodiment of the present application can be reduced by 50% based on the time consumption of the target training process of the traditional method, where 50% is the experimental result.

[0265] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0266] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0267] See also Figure 5 , Figure 5 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the model training device 1 comprises: a deployment unit 11, a processing unit 12, a transfer unit 13, an estimation unit 14, a loss determination unit 15, and an update unit 16. Among them:

[0268] A deployment unit 11 is used to deploy a first model in the running memory of i devices, where the first model is a model with natural language processing capability;

[0269] The deployment unit 11 is used to deploy a second model in the running memory of the m devices, where the second model is used to estimate the degree of improvement of the generated words to the target matching degree, where the target matching degree is the matching degree between the text generated by the first model and the preset requirements;

[0270] The deployment unit 11 is used to deploy the first model in the video memory of n devices, where n is the sum of i and m;

[0271] A processing unit 12, configured to process the training prompt words using the first model deployed in the video memory of the n devices to generate a first text;

[0272] A transfer unit 13, configured to transfer the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices;

[0273] The transfer unit 13 is used to transfer the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0274] The transfer unit 13 is used to transfer the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0275] An estimation unit 14 is used to process the training prompt words and the first text by using the second model in the video memory of the m devices, and estimate a first improvement degree of the target word in the first text to the target matching degree;

[0276] a loss determining unit 15, configured to determine a first loss of the first model based on the first lift, wherein the first lift is negatively correlated with the first loss;

[0277] The updating unit 16 is used to update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model.

[0278] In conjunction with any embodiment of the present application, the deployment unit 11 is specifically used to:

[0279] Divide the first model into n parts to obtain n first sub-models;

[0280] Deploy the n copies of the first sub-model in the video memory of the n devices, wherein the first sub-model corresponds to a device in the n devices one by one;

[0281] The processing unit 12 is specifically used for:

[0282] The training prompt words are processed by using the n first sub-models deployed in the video memory of the n devices to generate the first text.

[0283] In combination with any embodiment of the present application, the n devices include a shared device, the video memory of the shared device includes a shared storage space, the n devices can access the shared storage space, and the n first sub-models are stored in the shared storage space;

[0284] The processing unit 12 is specifically used for:

[0285] For each of the n devices, target data required for completing target processing of the corresponding first sub-model is obtained from the shared storage space, where the target processing is used to implement processing of the first text by the first model;

[0286] The n first sub-models deployed in the video memory of the n devices are used to process the corresponding target data to generate the first text.

[0287] In conjunction with any embodiment of the present application, the processing unit 12 is specifically configured to:

[0288] For each of the n devices, determining data required to complete the target processing of the corresponding first sub-model from the shared storage space as the target data;

[0289] For each of the n devices, the target data stored in the shared storage space is stored in a video memory of the device.

[0290] In conjunction with any embodiment of the present application, the deployment unit 11 is specifically used to:

[0291] Divide the first model into i parts to obtain i second sub-models;

[0292] Deploy the i copies of the second sub-model in the running memory of the i devices, wherein the second sub-model corresponds one-to-one to the devices in the i devices;

[0293] The transfer unit 13 is specifically used for:

[0294] Transferring the i copies of the second sub-model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices;

[0295] The updating unit 16 is specifically used for:

[0296] Based on the first loss, update the parameters of the i second sub-models in the video memory of the i devices to obtain i third sub-models;

[0297] The i third sub-models are merged to obtain the target model.

[0298] In combination with any embodiment of the present application, the transfer unit 13 is further used for:

[0299] The first training information stored in the running memory of the i devices is transferred to the video memory of the i devices, wherein the first training information includes: parameters of the second sub-model, the gradient of the first model in the previous round of training, and the current number of training rounds.

[0300] In combination with any embodiment of the present application, the i third sub-models are obtained through training of the current number of training rounds;

[0301] The updating unit 16 is specifically used for:

[0302] When the current training round number is the last training round number, the i third sub-models are merged to obtain the target model.

[0303] In combination with any embodiment of the present application, the updating unit 16 is further configured to:

[0304] When the current training round number is not the last training round number, transferring the i copies of the third sub-model from the video memory of the i devices to the running memory of the i devices;

[0305] Each of the n devices performs the step of merging the i copies of the third sub-model in the running memory of the i devices, so that the running memory of each of the n devices stores the updated first model, and the updated first model is obtained by merging the i copies of the third sub-model;

[0306] Each of the n devices executes the step of dividing the updated first model into n fourth sub-models, so that a copy of the fourth sub-model is stored in the running memory of each of the n devices, and the fourth sub-models stored in the running memories of different devices among the n devices are different;

[0307] Each of the n devices executes to store the fourth sub-model stored in the running memory into the shared storage space.

[0308] In conjunction with any embodiment of the present application, the deployment unit 11 is specifically used to:

[0309] Divide the second model into m parts to obtain m fifth sub-models;

[0310] Deploy the m copies of the fifth sub-model in the running memory of the m devices, wherein the fifth sub-model corresponds one-to-one to a device in the m devices;

[0311] The transfer unit 13 is specifically used for:

[0312] Transferring the m copies of the fifth sub-model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices;

[0313] The processing unit 12 is specifically used for:

[0314] The m fifth sub-models in the video memory of the m devices are used to process the training prompt words and the first text to estimate the first lift.

[0315] In combination with any embodiment of the present application, the transfer unit 13 is further used for:

[0316] The second training information stored in the running memory of the m devices is transferred to the video memory of the m devices, wherein the second training information includes: parameters of the fifth sub-model, the gradient of the second model in the previous round of training, the current number of training rounds, and the first training data of the fifth sub-model in the current number of training rounds, wherein the first training data is obtained by segmenting the training prompt words and the first text.

[0317] In combination with any embodiment of the present application, a third model is deployed in the video memory of the target device among the n devices, and the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0318] The estimation unit 14 is further used for:

[0319] Obtaining a second lift, where the second lift is an actual lift of the target word to the target matching degree, and the second lift is obtained by processing the training prompt word and the first text by the third model;

[0320] The above-mentioned loss determination unit 15 is further used for:

[0321] determining a second loss of the second model based on a first difference between the second lift and the first lift, wherein the second loss is positively correlated with the first difference;

[0322] The updating unit 16 is further used for:

[0323] Based on the second loss, update the parameters of the m fifth sub-models in the video memory of the m devices to obtain m sixth sub-models;

[0324] The m sixth sub-models are merged to obtain an updated second model.

[0325] In combination with any embodiment of the present application, the deployment unit 11 is further used for:

[0326] Deploy a third model in the running memory of a target device among the n devices, wherein the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement;

[0327] The transfer unit 13 is also used for:

[0328] Transferring the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device;

[0329] The estimation unit 14 is further used for:

[0330] Using the third model deployed in the video memory of the target device to process the training prompt word and the first text, a second lift is obtained, where the second lift is an actual lift of the target word to the target matching degree;

[0331] The above-mentioned loss determination unit 15 is specifically used for:

[0332] The first loss is determined based on a first difference between the second lift and the first lift, and the first loss is positively correlated with the first difference.

[0333] In an embodiment of the present application, the model training device deploys the first model in the running memory of i devices, the second model in the running memory of m devices, and the first model in the video memory of n devices, wherein the n devices include i devices and m devices. After the first model deployed in the video memory of n devices is used to process the training prompt word to generate the first text, the reasoning process of the first model is completed. Then the first model deployed in the video memory of n devices is transferred from the video memory of n devices to the running memory of n devices, thereby releasing the video memory in n devices. After the video memory in n devices is released, the first model deployed in the running memory of i devices is transferred from the running memory of i devices to the video memory of i devices, and the second model deployed in the running memory of m devices is transferred from the running memory of m devices to the video memory of m devices. Then, the second model in the video memory of m devices is used to process the training prompt word and the first text, and the first improvement degree of the target word in the first text to the target matching degree is estimated. The first loss of the first model is determined based on the first improvement degree, and the first improvement degree is negatively correlated with the first loss. Based on the first loss, the parameters of the first model in the video memory of i devices are updated to obtain the target model, thereby completing the training of the first model. Based on the technical solution provided in the embodiment of the present application, the reasoning of the first model and the training of the first model can be completed by n devices, and because the reasoning process of the first model and the training process of the first model are not performed simultaneously, the technical solution can alternately perform the reasoning process and the training process through n devices, thereby improving the utilization rate of device resources, thereby reducing the device resources required to complete the reasoning of the first model and the training of the first model.

[0334] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0335] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0336] The processor 21 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 21 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0337] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0338] The input device 23 is used to input data and / or signals, and the output device 24 is used to generate data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0339] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store the first model and the second model obtained through the input device 23, or the memory 22 can also be used to store the target model obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0340] Understandably, Figure 6 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0341] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0342] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0343] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0344] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0345] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0346] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0347] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A model training method, characterized in that: The method comprises: Deploy a first model in the running memory of i devices, where the first model is a model with natural language processing capability; Deploy a second model in the running memory of the m devices, where the second model is used to estimate the degree of improvement of the generated words to a target matching degree, where the target matching degree is the matching degree between the text generated by the first model and a preset requirement; Deploy the first model in the video memory of n devices, where the n devices include the i devices and the m devices; Using the first model deployed in the video memory of the n devices, the training prompt words are processed to generate a first text; Transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices; Transferring the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices; Transferring the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices; Using the second model in the video memory of the m devices, the training prompt word and the first text are processed to estimate a first improvement degree of the target word in the first text to the target matching degree; The i devices determine a first loss of the first model based on the first lift, where the first lift is negatively correlated with the first loss; The i devices update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model.

2. The method according to claim 1, characterized in that The deploying the first model in the video memory of n devices includes: Divide the first model into n parts to obtain n first sub-models; Deploy the n copies of the first sub-model in the video memory of the n devices, wherein the first sub-model corresponds to a device in the n devices one by one; The step of processing the training prompt words by using the first model deployed in the video memory of the n devices to generate the first text includes: The training prompt words are processed by using the n first sub-models deployed in the video memory of the n devices to generate the first text.

3. The method according to claim 2, characterized in that The n devices include a shared device, the video memory of the shared device includes a shared storage space, the n devices can all access the shared storage space, and the n first sub-models are stored in the shared storage space; The step of processing the training prompt words by using the n first sub-models deployed in the video memory of the n devices to generate the first text includes: For each of the n devices, target data required for completing target processing of the corresponding first sub-model is obtained from the shared storage space, wherein the target processing is used to implement processing of the training prompt word by the first model; The target processing is performed on the corresponding target data using the n first sub-models deployed in the video memories of the n devices to generate the first text.

4. The method according to claim 3, characterized in that For each of the n devices, the target data required to complete the target processing of the corresponding first sub-model is obtained from the shared storage space, including: For each of the n devices, determining data required to complete the target processing of the corresponding first sub-model from the shared storage space as the target data; For each of the n devices, the target data stored in the shared storage space is stored in a video memory of the device.

5. The method according to any one of claims 1 to 4, characterized in that: The deploying the first model in the running memory of the i devices includes: Divide the first model into i parts to obtain i second sub-models; Deploy the i copies of the second sub-model in the running memory of the i devices, wherein the second sub-model corresponds one-to-one to the devices in the i devices; The transferring the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices includes: Transferring the i copies of the second sub-model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices; The i devices update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model, including: The i devices update the parameters of the i second sub-models in the video memory of the i devices based on the first loss to obtain i third sub-models; The i devices merge the i third sub-models to obtain the target model.

6. The method according to claim 5, characterized in that The i third sub-models are obtained through training of the current training rounds; The i devices merge the i third sub-models to obtain the target model, including: When the current number of training rounds is the last number of training rounds, the i devices merge the i third sub-models to obtain the target model.

7. The method according to claim 6, characterized in that The method further comprises: When the current training round number is not the last training round number, transferring the i copies of the third sub-model from the video memory of the i devices to the running memory of the i devices; Each of the n devices performs the step of merging the i copies of the third sub-model in the running memory of the i devices, so that the running memory of each of the n devices stores the updated first model, and the updated first model is obtained by merging the i copies of the third sub-model; Each of the n devices executes the step of dividing the updated first model into n fourth sub-models, so that a copy of the fourth sub-model is stored in the running memory of each of the n devices, and the fourth sub-models stored in the running memories of different devices among the n devices are different; Each of the n devices executes to store the fourth sub-model stored in the running memory into the shared storage space.

8. The method according to any one of claims 1 to 4, characterized in that The deploying the second model in the running memory of the m devices includes: Divide the second model into m parts to obtain m fifth sub-models; Deploy the m copies of the fifth sub-model in the running memory of the m devices, wherein the fifth sub-model corresponds one-to-one to a device in the m devices; The transferring the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices includes: Transferring the m copies of the fifth sub-model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices; The using the second model in the video memory of the m devices to process the training prompt word and the first text, and estimating and generating a first improvement degree of the target word in the first text to the target matching degree, includes: The m fifth sub-models in the video memory of the m devices are used to process the training prompt words and the first text to estimate the first lift.

9. The method according to claim 8, characterized in that A third model is deployed in the video memory of the target device among the n devices, and the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement; After transferring the m copies of the fifth sub-models deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices, the method further includes: Obtaining a second lift, where the second lift is an actual lift of the target word to the target matching degree, and the second lift is obtained by processing the training prompt word and the first text by the third model; determining a second loss of the second model based on a first difference between the second lift and the first lift, wherein the second loss is positively correlated with the first difference; Based on the second loss, update the parameters of the m fifth sub-models in the video memory of the m devices to obtain m sixth sub-models; The m sixth sub-models are merged to obtain an updated second model.

10. The method according to any one of claims 1 to 4, characterized in that: Before transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices, the method further includes: Deploy a third model in the running memory of a target device among the n devices, wherein the third model is used to evaluate the matching degree between the text generated by the first model and the preset requirement; After transferring the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices, and before determining the first loss of the first model based on the first improvement, the method further includes: Transferring the third model deployed in the running memory of the target device from the running memory of the target device to the video memory of the target device; Using the third model deployed in the video memory of the target device to process the training prompt word and the first text, a second lift is obtained, where the second lift is an actual lift of the target word to the target matching degree; Using the second model in the video memory of the m devices, the training prompt word and the first text are processed to estimate a third improvement degree of a reference word in the first text to the target matching degree, wherein the reference word is a word preceding the target word; The determining a first loss of the first model based on the first lift includes: Determine an advantage function based on a second difference between the second lift and the third lift, the advantage function representing the improvement of the target matching degree when the target word is generated as the next word of the reference word when the reference word has been generated; Based on the advantage function, the first loss is determined, the first loss being positively correlated with the first difference.

11. A model training device, characterized in that: The model training device comprises: A deployment unit, configured to deploy a first model in a running memory of i devices, wherein the first model is a model having a natural language processing capability; The deployment unit is further used to deploy a second model in the running memory of the m devices, where the second model is used to estimate the degree of improvement of the generated words to the target matching degree, where the target matching degree is the matching degree between the text generated by the first model and the preset requirements; The deployment unit is further used to deploy the first model in the video memory of n devices, where n is the sum of i and m; A processing unit, configured to process the training prompt words using the first model deployed in the video memory of the n devices to generate a first text; A transfer unit, configured to transfer the first model deployed in the video memory of the n devices from the video memory of the n devices to the running memory of the n devices; The transfer unit is further used to transfer the first model deployed in the running memory of the i devices from the running memory of the i devices to the video memory of the i devices; The transfer unit is further used to transfer the second model deployed in the running memory of the m devices from the running memory of the m devices to the video memory of the m devices; an estimating unit, configured to process the training prompt words and the first text using the second model in the video memory of the m devices, and estimate a first improvement degree of the target word in the first text to the target matching degree; a loss determining unit, configured to determine a first loss of the first model based on the first lift, wherein the first lift is negatively correlated with the first loss; An updating unit is used to update the parameters of the first model in the video memory of the i devices based on the first loss to obtain a target model.

12. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises a computer program or instructions; when the computer program or instructions are run on a computer, the computer is caused to execute the method according to any one of claims 1 to 10.

Images