Data updating method, device and equipment applied to big language model training
By dividing the large language model into multiple model partitions, each partition only processes its corresponding model parameters, it solves the problem of computing resources and memory resource occupation caused by the large amount of model parameters, and achieves more efficient model training and hardware resource utilization.
Patent Information
- Application Number
- CN202411704161.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2025-03-07
AI Technical Summary
In training of large language models, due to the large amount of model parameters, the model gradients and data transmission and synchronization between model parameters of different model partitions occupy a lot of computing resources and memory resources, which limits the improvement of training efficiency and leads to low utilization of hardware resources of computer equipment.
By dividing the target data model into several model partitions, each model partition contains only part of the model parameters of the target data model. For each model partition, the data to be processed is calculated based on the model parameters contained in the model partition to obtain the partition gradient, and the global gradient update model parameters are obtained based on several partition gradients.
The model gradients and data transmission and synchronization between model parameters of different model partitions are reduced, thereby reducing redundant calculation and communication overhead, improving the utilization rate of hardware resources, and improving model training efficiency.
Smart Images
Figure CN119578488A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data updating method, apparatus, computer equipment, storage medium and computer program product for large language model training. Background Art
[0002] With the rapid development of deep learning technology and hardware technology, the application of large models has become more and more common. For example, large language models are machine learning models with super-large parameters (usually more than one billion) and super computing resources. They can process massive amounts of data and complete various complex tasks, such as natural language processing and image recognition.
[0003] When dealing with massive amounts of data and complex models, the related distributed model training strategies can solve the problems caused by memory and computing resources to a certain extent. However, due to the large number of model parameters, the redundant computing and communication overhead caused by data transmission and synchronization between model gradients and model parameters of different model partitions occupy more computing resources and memory resources, limiting the improvement of training efficiency and leading to low utilization of computer equipment hardware resources. Summary of the invention
[0004] Based on this, it is necessary to provide a data updating method, device, computer equipment, computer-readable storage medium and computer program product for large language model training, which can improve the utilization rate of computer equipment hardware resources during model training, in order to solve the above technical problems.
[0005] In a first aspect, the present application provides a data updating method applied to large language model training, comprising:
[0006] Obtain the data to be processed and the target data model;
[0007] Dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a portion of model parameters of the target data model;
[0008] For each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain a partition gradient;
[0009] A global gradient is obtained according to the plurality of partition gradients, and the model parameters are updated according to the global gradient.
[0010] In one embodiment, after obtaining the data to be processed, the method further includes:
[0011] Dividing the data to be processed into a plurality of input data subsets;
[0012] Allocating different subsets of the input data to different computing nodes; wherein each computing node corresponds to one of the model partitions;
[0013] Accordingly, for each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain several partition gradients, including:
[0014] On each of the computing nodes, the input data subset is calculated according to the model parameters included in the corresponding model partition to obtain a partition gradient.
[0015] In one embodiment, each of the model partitions corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters contained in the model partition; on each of the computing nodes, the input data subset is calculated according to the model parameters contained in the corresponding model partition to obtain a partition gradient, including:
[0016] On each of the computing nodes, the input data subsets are calculated in parallel according to the mapping relationship to obtain a partition gradient corresponding to each of the model partitions.
[0017] In one embodiment, the method further comprises:
[0018] Obtaining optimizer status for the model partition;
[0019] A mapping relationship between the partition index, the optimizer state, and the model parameters included in the model partition is established.
[0020] In one embodiment, the step of calculating the input data subset on each computing node according to the model parameters included in the corresponding model partition to obtain a partition gradient includes:
[0021] On each of the computing nodes, inputting the input data subset into the corresponding model partition to obtain target output data;
[0022] Calculating the partition loss value of the model partition according to the target output data and a preset loss function;
[0023] A partition gradient is calculated based on the partition loss value and the model parameters contained in the model partition.
[0024] In one embodiment, obtaining a global gradient according to the plurality of partition gradients comprises:
[0025] The average value of several partition gradients is calculated to obtain the global gradient.
[0026] In a second aspect, the present application also provides a data updating device for large language model training, comprising:
[0027] A data acquisition module is used to acquire the data to be processed and the target data model;
[0028] A model partitioning module, used for dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a part of the model parameters of the target data model;
[0029] A gradient calculation module, used for calculating the data to be processed according to the model parameters contained in the model partition for each of the model partitions to obtain a partition gradient;
[0030] A data updating module is used to obtain a global gradient according to a plurality of partition gradients, and to update the model parameters according to the global gradient.
[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0032] Obtain the data to be processed and the target data model;
[0033] Dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a portion of model parameters of the target data model;
[0034] For each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain a partition gradient;
[0035] A global gradient is obtained according to the plurality of partition gradients, and the model parameters are updated according to the global gradient.
[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:
[0037] Obtain the data to be processed and the target data model;
[0038] Dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a portion of model parameters of the target data model;
[0039] For each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain a partition gradient;
[0040] A global gradient is obtained according to the plurality of partition gradients, and the model parameters are updated according to the global gradient.
[0041] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:
[0042] Obtain the data to be processed and the target data model;
[0043] Dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a portion of model parameters of the target data model;
[0044] For each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain a partition gradient;
[0045] A global gradient is obtained according to the plurality of partition gradients, and the model parameters are updated according to the global gradient.
[0046] The above-mentioned data update method, device, computer equipment, storage medium and computer program product applied to large language model training obtain the data to be processed and the target data model, and divide the target data model into several model partitions, so that each model partition only contains a part of the model parameters of the target data model. For each model partition, the data to be processed is calculated according to the model parameters contained in the model partition to obtain the partition gradient, thereby ensuring that each model partition only calculates the part of the model parameters contained in the model partition, and obtains the partition gradient uniquely corresponding to this model partition, without saving the model parameters of other parts. Finally, according to several partition gradients, a global gradient is obtained, and the model parameters are updated according to the global gradient. Therefore, the above-mentioned method greatly reduces the data transmission and synchronization between the model gradients and model parameters of different model partitions, thereby reducing redundant calculations and communication overheads. In scenarios where the number of model parameters is large, there is no need to occupy more computing resources and memory resources, thereby improving the model training efficiency and improving the utilization rate of computer equipment hardware resources during model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1An application environment diagram of a data updating method applied to large language model training in one embodiment;
[0049] Figure 2 A schematic diagram of a flow chart of a data updating method applied to large language model training in one embodiment;
[0050] Figure 3 This is a flow chart of step S206 of a data updating method applied to large language model training in one embodiment;
[0051] Figure 4 It is a structural block diagram of a data updating device applied to large language model training in one embodiment;
[0052] Figure 5 is an internal structure diagram of a computer device in one embodiment;
[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0055] The data updating method for large language model training provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network, and the terminal 102 can be used to obtain the data to be processed and send the data to be processed to the server 104. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The data storage system can be used to store information such as the data to be processed and the model parameters corresponding to each model partition, and can also be used to store data such as the calculated partition gradient. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.
[0056] In an exemplary embodiment, Figure 2 As shown, a data updating method for large language model training is provided, and the method is applied to Figure 1The server 104 in the example is used as an example to illustrate the method, which includes the following steps S202 to S208. Among them:
[0057] Step S202, obtaining the data to be processed and the target data model.
[0058] Exemplarily, the server 104 may divide the data to be processed into several input data subsets, and assign different input data subsets to different computing nodes; wherein each computing node corresponds to a model partition. Dividing the target data model into multiple regions, storing them in different memory regions, and assigning different computer processor nodes for computing can improve the utilization efficiency of multi-core devices, thereby improving the efficiency of model training.
[0059] Among them, each model partition corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters contained in the model partition. Model partitioning is a method of dividing a data table into multiple subsets according to certain rules. Through partitioning, the data in a large table can be split into multiple smaller parts, such as by time, region, or business logic, so as to reduce the amount of data in a single partition and allow queries to be performed more quickly within the specified partition without having to traverse the entire data table. The partition index is an index established for each partition. Since each partition is a relatively independent data set, in order to further optimize the query, the system can create a separate index on each partition.
[0060] Step S204: divide the target data model into a plurality of model partitions.
[0061] Each model partition contains a portion of model parameters of the target data model.
[0062] Among them, model parameters refer to the trainable weights and biases in the model. For example, in a neural network, model parameters include the weights of each layer of neuron connections and the bias of each neuron. These parameters are constantly adjusted during the model training process to make the model's predictions as accurate as possible. The optimizer state is used to record and manage some additional information during the parameter update process. During the training process, the optimizer adjusts the model parameters according to the gradient of each step, and the optimizer state records the intermediate data of this adjustment.
[0063] For example, the server 104 may formulate a model partitioning scheme based on the model structure and computing resources (such as the number of processors). Each partition should contain a portion of the model parameters and correspond to a portion of the input data and computing tasks. The weight of each partition is recorded as ,in, 1,N}. This step can support model parallel strategies, such as tensor parallel strategies, to further reduce video memory.
[0064] Step S206: For each model partition, the data to be processed is calculated according to the model parameters contained in the model partition to obtain the partition gradient.
[0065] Exemplarily, when the server 104 divides the data to be processed into several input data subsets, the input data subset can be calculated on each computing node according to the model parameters included in the corresponding model partition to obtain the partition gradient.
[0066] Exemplarily, the server 104 can calculate the input data subset in parallel on each computing node according to the mapping relationship to obtain the partition gradient corresponding to each model partition. Further, the server 104 can also obtain the optimizer state of the model partition, establish the mapping relationship between the partition index, the optimizer state and the model parameters contained in the model partition, and establish a clear mapping relationship between the gradient, the model parameters and the optimizer state in advance in the model partition stage, which can significantly reduce memory usage and communication overhead.
[0067] Furthermore, the server 104 may, for each partition in the model parameters , assign a globally unique number or index, then the server 104 can initialize a mapping table , which is used to store parameter numbers, their partition indexes, and the corresponding optimizer states. The mapping table can be a two-dimensional array or dictionary data structure, where each row or key-value pair represents a parameter and its related information. For example, a mapping table represented by a dictionary can be as follows, with dimensions , is the number of parameters.
[0068] For example, the server 104 can perform forward propagation and backward propagation in parallel on each computing node according to the model partition using the corresponding model parameters, thereby calculating the output of the model, and calculating the gradient according to the loss function, and storing the gradient on each partition. When calculating the gradient, the server 104 needs to ensure that the model parameters of a partition are saved on each computing node. , i is the partition number, other partitions are discarded, and according to the model mapping table Find the corresponding mapping relationship. Since the mapping relationship has been established, the gradient on each partition is associated with its corresponding model parameter.
[0069] Step S208, obtaining a global gradient based on several partition gradients, and updating the model parameters based on the global gradient.
[0070] Exemplarily, the server 104 can calculate the average value of several partition gradients to obtain the global gradient. In the related scheme, each computing node is responsible for calculating the part assigned to it, but each computing node will generate the complete gradient of the entire model after calculation. Each computing node not only calculates its own partition, but also obtains updates of other partitions in multiple rounds of exchanges to keep the whole model synchronized, which requires multiple rounds of data exchange between computing nodes to maintain consistency. Therefore, in the traditional full model synchronization method, two rounds of communication are usually required: one for calculating the preliminary summary and the other for confirming the final full model synchronization. Each round of exchange has data dependency. In this method, each computing node only focuses on calculating and saving its own partition, without having to calculate the gradient of the entire model, eliminating the dependency of full model synchronization: each computing node is only responsible for its own assigned parameters, and sends the calculation results to other computing nodes through an all-gather, and other computing nodes only need their own partition results, which no longer requires additional intermediate synchronization. Through a one-time exchange, all computing nodes can fully obtain the data of other partitions and then splice them into updates of the full model. After this single operation, each computing node has complete global information and does not need to be confirmed again.
[0071] In the above data update method applied to large language model training, by obtaining the data to be processed and the target data model, and dividing the target data model into several model partitions, each model partition only contains a part of the model parameters of the target data model. For each model partition, the data to be processed is calculated according to the model parameters contained in the model partition to obtain the partition gradient, thereby ensuring that each model partition only calculates the part of the model parameters contained in the model partition, and obtains the partition gradient uniquely corresponding to this model partition, without saving the model parameters of other parts. Finally, based on several partition gradients, a global gradient is obtained, and the model parameters are updated according to the global gradient. The gradient, model parameters and optimizer state are divided according to the corresponding relationship to ensure that the gradient on each computing node directly corresponds to its locally stored model parameters and optimizer state. Therefore, the above method greatly reduces the data transmission and synchronization between the model gradients and model parameters of different model partitions, thereby reducing redundant calculations and communication overheads, avoiding frequent transmission and synchronization of data between nodes, and thus reducing communication overheads. At the same time, since each computing node only processes data and parameters related to it, memory usage is reduced. In scenarios with a large number of model parameters, there is no need to occupy more computing resources and memory resources, which improves the utilization of hardware resources, improves model training efficiency, and improves the utilization of computer equipment hardware resources during model training.
[0072] In an exemplary embodiment, Figure 3 As shown, step S206 includes steps S302 to S306. Among them:
[0073] Step S302: On each computing node, input the input data subset into the corresponding model partition to obtain target output data.
[0074] For example, the server 104 needs to ensure that the model parameters of a partition are stored on each computing node. , i is the partition number, other partitions are discarded, and according to the model mapping table Find the corresponding mapping relationship, input the input data subset into the corresponding model partition, and obtain the target output data.
[0075] Step S304, calculating the partition loss value of the model partition according to the target output data and the preset loss function.
[0076] For example, the server 104 may use a loss function (such as mean square error, cross entropy, etc.) to calculate the predicted output of the model ( ) and the true label ( ) to get the loss value ( ). For the entire dataset (or batch), the total loss is some aggregation (such as an average) of the losses of the individual partitions.
[0077]
[0078]
[0079] Step S306, calculating the partition gradient according to the partition loss value and the model parameters contained in the model partition.
[0080] Exemplarily, the server 104 can update the model parameters on each partition according to the gradient, learning rate, and optimizer state. Due to the existence of the mapping relationship, each partition can directly access and update its corresponding model parameters and optimizer state, which can significantly reduce communication and memory overhead. The server 104 can calculate the gradient of the loss function to the model parameters through the back propagation algorithm. This involves the application of the chain rule, starting from the output layer, and calculating the gradient layer by layer. For each partition, the server 104 can calculate the gradient of the loss ((L_i)) to the partition parameters .in,
[0081]
[0082] Next, the server 104 can store the calculated gradients on each partition. Since the mapping relationship has been established, the partition gradients on each partition are associated with its corresponding model parameters. The server 104 can aggregate the partition gradients calculated on multiple partitions to form a global gradient update, which is achieved by averaging the partition gradients of each partition. The formula is as follows:
[0083]
[0084] Finally, the server 104 can use an optimization algorithm (such as SGD, Adam, etc.) and the aggregated gradients to update the model parameters. is the learning rate, which controls the step size of parameter updates.
[0085]
[0086] In the above embodiment, forward propagation and back propagation calculations can be performed simultaneously on multiple computing nodes, thereby significantly improving the training speed. In addition, since each computing node only processes data and parameters related to it, local calculation and storage of gradients are achieved, ensuring the accuracy and efficiency of gradient calculations, and avoiding frequent transmission and synchronization of gradients between nodes, thereby reducing communication overhead and further improving computing efficiency.
[0087] In another exemplary embodiment, the server 104 obtains the data to be processed and the target data model, divides the data to be processed into several input data subsets, and allocates different input data subsets to different computing nodes, each computing node corresponds to a model partition, each model partition corresponds to a partition index, and there is a mapping relationship between the partition index, optimizer state, and model parameters contained in the model partition.
[0088] Next, the server 104 divides the target data model into several model partitions; each model partition contains a part of the model parameters of the target data model. On each computing node, a subset of the input data is input into the corresponding model partition to obtain the target output data. The partition loss value of the model partition is calculated based on the target output data and a preset loss function. The partition gradient is calculated based on the partition loss value and the model parameters contained in the model partition.
[0089] Finally, the server 104 calculates the average of several partition gradients to obtain a global gradient, and updates the model parameters according to the global gradient.
[0090] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0091] Based on the same inventive concept, the embodiment of the present application also provides a data updating device for large language model training for implementing the data updating method for large language model training involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the data updating device for large language model training provided below can refer to the limitations of the data updating method for large language model training above, and will not be repeated here.
[0092] In an exemplary embodiment, Figure 4 As shown, a data updating device for large language model training is provided, including: a data acquisition module 402, a model division module 404, a gradient calculation module 406 and a data updating module 408, wherein:
[0093] The data acquisition module 402 is used to acquire the data to be processed and the target data model;
[0094] A model partitioning module 404 is used to partition the target data model into a plurality of model partitions; wherein each model partition contains a portion of model parameters of the target data model;
[0095] A gradient calculation module 406 is used to calculate the data to be processed according to the model parameters contained in the model partition for each model partition to obtain a partition gradient;
[0096] The data updating module 408 is used to obtain a global gradient according to a plurality of partition gradients, and update the model parameters according to the global gradient.
[0097] In one embodiment, the device further comprises:
[0098] Data sharding module, used to divide the data to be processed into several input data subsets;
[0099] A node allocation module, used for allocating different input data subsets to different computing nodes; wherein each computing node corresponds to a model partition;
[0100] Accordingly, the gradient calculation module 406 is specifically used to: on each computing node, calculate the input data subset according to the model parameters included in the corresponding model partition to obtain the partition gradient.
[0101] In one embodiment, each model partition corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters included in the model partition; the gradient calculation module 406 includes:
[0102] The partition gradient calculation unit is used to calculate the input data subset in parallel on each computing node according to the mapping relationship to obtain the partition gradient corresponding to each model partition.
[0103] In one embodiment, the device further comprises:
[0104] The optimizer data acquisition module is used to obtain the optimizer status of the model partition;
[0105] The mapping module is used to establish the mapping relationship between the partition index, the optimizer state, and the model parameters contained in the model partition.
[0106] In one of the embodiments, the partition gradient calculation unit is specifically used to: on each computing node, input a subset of input data into a corresponding model partition to obtain target output data; calculate a partition loss value of the model partition based on the target output data and a preset loss function; calculate a partition gradient based on the partition loss value and the model parameters contained in the model partition.
[0107] In one embodiment, the data updating module 408 is specifically used to calculate an average value of several partition gradients to obtain a global gradient.
[0108] Each module in the data updating device for large language model training can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0109] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information such as data to be processed and model parameters corresponding to each model partition, and can also be used to store data such as calculated partition gradients. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data update method applied to large language model training is implemented.
[0110] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data update method applied to large language model training is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0111] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0112] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining data to be processed and a target data model; dividing the target data model into a plurality of model partitions; wherein each model partition contains a part of the model parameters of the target data model; for each model partition, calculating the data to be processed according to the model parameters contained in the model partition to obtain a partition gradient; obtaining a global gradient according to the plurality of partition gradients, and updating the model parameters according to the global gradient.
[0113] In one embodiment, when the processor executes the computer program, the following steps are also implemented: dividing the data to be processed into several input data subsets; allocating different input data subsets to different computing nodes; wherein each computing node corresponds to a model partition; on each computing node, calculating the input data subset according to the model parameters contained in the corresponding model partition to obtain a partition gradient.
[0114] In one embodiment, when the processor executes the computer program, the following steps are also implemented: each model partition corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters contained in the model partition; on each computing node, according to the mapping relationship, a subset of the input data is calculated in parallel to obtain a partition gradient corresponding to each model partition.
[0115] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the optimizer state of the model partition; and establishing a mapping relationship between the partition index, the optimizer state, and the model parameters included in the model partition.
[0116] In one embodiment, when the processor executes the computer program, the following steps are also implemented: on each computing node, a subset of the input data is input into the corresponding model partition to obtain target output data; based on the target output data and a preset loss function, a partition loss value of the model partition is calculated; based on the partition loss value and the model parameters contained in the model partition, a partition gradient is calculated.
[0117] In one embodiment, when the processor executes the computer program, the following steps are further implemented: calculating an average value of several partition gradients to obtain a global gradient.
[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining data to be processed and a target data model; dividing the target data model into a plurality of model partitions; wherein each model partition contains a portion of model parameters of the target data model; for each model partition, calculating the data to be processed according to the model parameters contained in the model partition to obtain a partition gradient; obtaining a global gradient according to the plurality of partition gradients, and updating the model parameters according to the global gradient.
[0119] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: dividing the data to be processed into several input data subsets; allocating different input data subsets to different computing nodes; wherein each computing node corresponds to a model partition; on each computing node, calculating the input data subset according to the model parameters contained in the corresponding model partition to obtain a partition gradient.
[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: each model partition corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters contained in the model partition; on each computing node, according to the mapping relationship, a subset of the input data is calculated in parallel to obtain a partition gradient corresponding to each model partition.
[0121] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the optimizer state of the model partition; and establishing a mapping relationship between the partition index, the optimizer state, and the model parameters contained in the model partition.
[0122] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: on each computing node, a subset of the input data is input into the corresponding model partition to obtain the target output data; based on the target output data and a preset loss function, a partition loss value of the model partition is calculated; based on the partition loss value and the model parameters contained in the model partition, a partition gradient is calculated.
[0123] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: calculating an average value of several partition gradients to obtain a global gradient.
[0124] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining data to be processed and a target data model; dividing the target data model into a plurality of model partitions; wherein each model partition contains a portion of model parameters of the target data model; for each model partition, calculating the data to be processed according to the model parameters contained in the model partition to obtain a partition gradient; obtaining a global gradient based on the plurality of partition gradients, and updating the model parameters according to the global gradient.
[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: dividing the data to be processed into several input data subsets; allocating different input data subsets to different computing nodes; wherein each computing node corresponds to a model partition; on each computing node, calculating the input data subset according to the model parameters contained in the corresponding model partition to obtain a partition gradient.
[0126] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: each model partition corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters contained in the model partition; on each computing node, according to the mapping relationship, a subset of the input data is calculated in parallel to obtain a partition gradient corresponding to each model partition.
[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the optimizer state of the model partition; and establishing a mapping relationship between the partition index, the optimizer state, and the model parameters contained in the model partition.
[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: on each computing node, a subset of the input data is input into the corresponding model partition to obtain the target output data; based on the target output data and a preset loss function, a partition loss value of the model partition is calculated; based on the partition loss value and the model parameters contained in the model partition, a partition gradient is calculated.
[0129] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: calculating an average value of several partition gradients to obtain a global gradient.
[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0131] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data updating method for large language model training, characterized in that: The method comprises: Obtain the data to be processed and the target data model; Dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a portion of model parameters of the target data model; For each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain a partition gradient; A global gradient is obtained according to the plurality of partition gradients, and the model parameters are updated according to the global gradient.
2. The method according to claim 1, characterized in that: After obtaining the data to be processed, the method further includes: Dividing the data to be processed into a plurality of input data subsets; Allocating different subsets of the input data to different computing nodes; wherein each computing node corresponds to one of the model partitions; Accordingly, for each of the model partitions, the data to be processed is calculated according to the model parameters contained in the model partition to obtain several partition gradients, including: On each of the computing nodes, the input data subset is calculated according to the model parameters included in the corresponding model partition to obtain a partition gradient.
3. The method according to claim 2, characterized in that Each of the model partitions corresponds to a partition index, and there is a mapping relationship between the partition index and the model parameters included in the model partition; The step of calculating the input data subset on each of the computing nodes according to the model parameters included in the corresponding model partition to obtain a partition gradient includes: On each of the computing nodes, the input data subsets are calculated in parallel according to the mapping relationship to obtain a partition gradient corresponding to each of the model partitions.
4. The method according to claim 3, characterized in that: The method further comprises: Obtaining optimizer status for the model partition; A mapping relationship between the partition index, the optimizer state, and the model parameters included in the model partition is established.
5. The method according to claim 2, characterized in that: The step of calculating the input data subset on each of the computing nodes according to the model parameters included in the corresponding model partition to obtain a partition gradient includes: On each of the computing nodes, inputting the input data subset into the corresponding model partition to obtain target output data; Calculating the partition loss value of the model partition according to the target output data and a preset loss function; A partition gradient is calculated based on the partition loss value and the model parameters contained in the model partition.
6. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a global gradient based on the plurality of partition gradients includes: The average value of several partition gradients is calculated to obtain the global gradient.
7. A data updating device for large language model training, characterized in that: The device comprises: A data acquisition module is used to acquire the data to be processed and the target data model; A model partitioning module, used for dividing the target data model into a plurality of model partitions; wherein each of the model partitions contains a part of the model parameters of the target data model; A gradient calculation module, used for calculating the data to be processed according to the model parameters contained in the model partition for each of the model partitions to obtain a partition gradient; A data updating module is used to obtain a global gradient according to a plurality of partition gradients, and to update the model parameters according to the global gradient.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.