Model updating method and device, electronic device, medium and product

Abstract

The present disclosure provides a model updating method and device, electronic equipment, medium and product. The method comprises: before a first device performs a parameter updating operation of a current iteration round through a model updating task, loading first state data stored in a second device to the first device through a data loading task; wherein the execution process of the data loading task and the process of performing back propagation of the current iteration round of the model updating task overlap with each other; after the model updating task performs the parameter updating operation of the current iteration round, unloading second state data stored in the first device to the second device through a data unloading task; wherein the execution process of the data unloading task and the process of performing forward propagation of the next iteration round of the model updating task overlap with each other. This way can avoid the problem that the model updating process is stalled due to I / O waiting caused by the synchronous loading / unloading mechanism, thereby greatly improving the model updating efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a model update method and apparatus, electronic equipment, medium, and product. Background Technology

[0002] In traditional deep learning model training, model update refers to the process of adjusting model parameters through optimization algorithms to minimize the loss function. A typical training process includes three main stages: forward propagation, backward propagation, and parameter update.

[0003] In related technologies, model updates typically follow these steps: (1) Forward propagation: The input data is processed by the model to calculate the predicted value and obtain the loss function value. (2) Backward propagation: The gradient of the loss function with respect to the model parameters is calculated. (3) Parameter update: The optimizer adjusts the model parameters based on the gradient. The above process can be executed sequentially in each training batch, meaning that parameter updates can only be performed after forward and back propagation are completed.

[0004] However, in large-scale model training or resource-constrained environments, the above approach may suffer from a GPU memory bottleneck. For example, optimizer states are typically stored in GPU memory, and their storage overhead is proportional to the number of model parameters, leading to insufficient GPU memory during large model training. Correspondingly, when GPU memory is insufficient, some states need to be unloaded to other devices such as CPU memory for storage. However, since existing methods typically employ synchronous loading / unloading mechanisms, the model update process is stalled due to I / O wait, thus affecting the model update efficiency. Summary of the Invention

[0005] This disclosure provides a model update method, apparatus, electronic device, medium, and product to improve model update efficiency.

[0006] In a first aspect, this disclosure provides a model update method, the method comprising:

[0007] The first state data stored in the second device is loaded into the first device through a data loading task; wherein, the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially perform forward propagation, backpropagation and parameter update operations in each iteration round.

[0008] Based on the first state data, the parameter update operation for the current iteration is performed through the model update task;

[0009] The second state data stored in the first device is unloaded to the second device through a data unloading task; wherein the execution process of the data unloading task coincides with the forward propagation process of the model update task in the next iteration round; and the second state data is determined according to the parameter update result of the current iteration round and is used to perform the parameter update operation in the next iteration round.

[0010] Secondly, this disclosure provides a data processing method, including:

[0011] The data to be processed is input into the target data processing model to obtain the data processing result corresponding to the data to be processed; wherein, the data to be processed is any one of the following: text to be processed, speech to be processed, image to be processed, and video to be processed;

[0012] The target data processing model is obtained using the model update method described above.

[0013] Thirdly, this disclosure provides a model updating apparatus, the apparatus comprising:

[0014] A loading module is adapted to load first state data stored in a second device into the first device via a data loading task; wherein the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially execute forward propagation, backpropagation, and parameter update operations in each iteration round.

[0015] The update module is adapted to perform the parameter update operation of the current iteration round through the model update task based on the first state data;

[0016] The unloading module is adapted to unload second state data stored in the first device to the second device through a data unloading task; wherein the execution process of the data unloading task coincides with the forward propagation process of the model update task executing the next iteration round; and the second state data is determined according to the parameter update result of the current iteration round and is used to execute the parameter update operation of the next iteration round.

[0017] Fourthly, this disclosure provides a data processing apparatus, comprising:

[0018] The data processing module is adapted to input the data to be processed into the target data processing model to obtain the data processing result corresponding to the data to be processed; wherein, the data to be processed is any one of the following: text to be processed, speech to be processed, image to be processed, and video to be processed;

[0019] The target data processing model is obtained based on the aforementioned model update device.

[0020] Fifthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0021] In a sixth aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described method when executed by a processor / processor core.

[0022] In a seventh aspect, this disclosure provides a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in an electronic device, a processor in the electronic device performs the above-described method.

[0023] In the embodiments provided in this disclosure, on the one hand, a data loading task loads the first state data stored in the second device to the first device, thereby making the execution process of the data loading task coincide with the backpropagation process of the model update task in the current iteration round; on the other hand, a data unloading task unloads the second state data stored in the first device to the second device, thereby making the execution process of the data unloading task coincide with the forward propagation process of the model update task in the next iteration round. Therefore, in this method, a data loading task and a data unloading task are set up independently of the model update task, and the data loading task and the data unloading task can be executed asynchronously to coincide with the execution timing of the model update task, thereby avoiding the problem of the model update process stalling due to I / O waiting caused by the synchronous loading / unloading mechanism, and thus significantly improving the model update efficiency.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0026] Figure 1 A schematic flowchart of a model update method provided in one embodiment of this disclosure is shown;

[0027] Figure 2 A schematic flowchart of a model update method provided in another embodiment of this disclosure is shown;

[0028] Figure 3 A block diagram of a model update apparatus provided in an embodiment of this disclosure;

[0029] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0030] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0032] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0034] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0035] Figure 1 A schematic flowchart of a model update method provided in an embodiment of this disclosure is shown. This method is applied to a first device. Figure 1 As shown, the method includes:

[0036] Step S110: Load the first state data stored in the second device to the first device through the data loading task; wherein, the execution process of the data loading task coincides with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially execute forward propagation, backpropagation and parameter update operations in each iteration round.

[0037] Specifically, before the first device performs the parameter update operation for the current iteration through the model update task, the first state data stored in the second device can be loaded into the first device through the data loading task.

[0038] The first state data can be determined based on the parameter update result of the previous iteration and is used to perform the parameter update operation in the current iteration. Additionally, in the first iteration, the first state data can be pre-configured initial state data.

[0039] The first device is a computing device used to perform model updates, responsible for processes such as forward / backward propagation and parameter updates. For example, the first device can be a GPU, video memory, or other similar devices. The second device is used to store state data and can support dynamic unloading. For example, the second device can be a CPU, memory, or other similar devices.

[0040] The state data can include various types of data required during the parameter update process, such as optimizer state data and model parameter state data. Correspondingly, the first state data can include the optimizer state after the previous training round and / or high-precision parameter copies. Therefore, the first state data refers to the state data stored in the second device and determined based on the parameter update results of the previous iteration.

[0041] The model update task is used to implement model update-related operations, specifically performing forward propagation, backward propagation, and parameter updates, and can be executed through the computation stream (default stream). Correspondingly, the data loading task is used to load first-state data from the second device and can be executed through a dedicated CUDA stream (such as stream1). The data loading task can be executed asynchronously and in parallel with the model update task.

[0042] In this step, because a data loading task is set up to run concurrently with the model update task, the loading of the first state data can be triggered before the model update task performs its parameter update operation in the current iteration. Furthermore, the execution of the data loading task overlaps with the backpropagation process of the model update task in its current iteration. Therefore, in this step, the data loading task can be executed asynchronously and in parallel during the backpropagation of the model update task in its current iteration, allowing the data loading process to begin earlier.

[0043] Step S120: Based on the first state data, perform the parameter update operation for the current iteration through the model update task.

[0044] The parameter update operation can include updating various parameters such as optimizer state parameters and / or model parameters.

[0045] Step S130: The second state data stored in the first device is unloaded to the second device through the data unloading task; wherein, the execution process of the data unloading task coincides with the forward propagation process of the model update task in the next iteration round; and the second state data is determined according to the parameter update result of the current iteration round and is used to perform the parameter update operation in the next iteration round.

[0046] In this process, after the model update task performs the parameter update operation for the current iteration, the second state data stored in the first device is unloaded to the second device through the data unloading task. The execution process of the data unloading task coincides with the forward propagation process of the model update task for the next iteration. Furthermore, the second state data is determined based on the parameter update result of the current iteration and is used to perform the parameter update operation for the next iteration.

[0047] The second state data can be data such as the optimizer state and parameter copies after the current round of updates, for use in the next round. Therefore, the second state data refers to the state data stored in the first device and determined based on the parameter update results of the current iteration round.

[0048] It should be noted that during the model update process, model update processing needs to be performed based on the first state data loaded from the second device. Furthermore, after the model update, the first state data loaded from the second device can also be updated accordingly. Correspondingly, the second state data stored in the first device can include state data obtained based on the update result of the first state data. Thus, the second state data in the first device corresponding to the current iteration is the same as the first state data in the second device corresponding to the next iteration. In other words, the first state data in the second device corresponding to the current iteration is the same as the second state data in the first device corresponding to the previous iteration.

[0049] The data offloading task is used to offload the second state data from the first device to the second device, and can be executed through a dedicated CUDA stream (such as stream2). The data offloading task can also be executed asynchronously and in parallel with the model update task.

[0050] In this step, because a data unloading task is set up to run in parallel with the model update task, the unloading process of the second-state data can overlap with the forward propagation process of the model update task executing the next iteration. Therefore, in this step, the forward propagation process of the next iteration can be executed through the model update task during the execution of the data unloading task, thus avoiding the problem of delaying the start time of the next iteration due to the data unloading process.

[0051] In summary, the above method, through asynchronous loading / unloading, allows the loading process of state data to overlap with the backpropagation process, and the unloading process to overlap with the forward propagation process, thereby hiding communication latency. Furthermore, an iterative update chain can be formed between the first state data (historical results) and the second state data (current results) to ensure training continuity. Traditional synchronous loading / unloading schemes are prone to problems such as training process blockage and low hardware utilization. In the above method, on the one hand, before the first device performs parameter updates for the current iteration through the model update task, the first state data stored in the second device is loaded onto the first device via the data loading task, thus overlapping the execution process of the data loading task with the backpropagation process of the model update task for the current iteration; on the other hand, after the model update task performs parameter updates for the current iteration, the second state data stored in the first device is unloaded onto the second device via the data unloading task, thus overlapping the execution process of the data unloading task with the forward propagation process of the model update task for the next iteration. Therefore, this approach sets up data loading and data unloading tasks that are independent of the model update task. Furthermore, the data loading and data unloading tasks can be executed asynchronously, coinciding with the execution time of the model update task. This avoids the problem of the model update process stalling due to I / O waiting caused by the synchronous loading / unloading mechanism, thereby significantly improving the efficiency of model update.

[0052] Furthermore, those skilled in the art can make various modifications and variations to the embodiments disclosed herein:

[0053] I. Improvement plans for loading and / or unloading timing:

[0054] In one alternative implementation, the model update method can be based on multiple sample batches. These multiple sample batches (micro-batches) can be obtained by splitting the complete training batch into several smaller batches, sequentially performing forward / backward propagation, and accumulating gradients. Correspondingly, in the case of multiple batches, the data loading task can be initiated after the forward propagation process of the last sample batch has ended. The end of the forward propagation process means that the output of the last layer of the last micro-batch has been computed. By initiating the data loading task after the forward propagation of the last micro-batch has been completed, the problem of premature loading consuming GPU memory can be avoided, and the time for backpropagation can be effectively utilized for data loading.

[0055] Accordingly, the data unloading task can be initiated after the parameter update operations for multiple sample batches have been completed. Since the unloading task starts after all batches have been updated, parameter consistency is ensured. The completion of parameter update operations means that gradient accumulation for all micro-batches has been completed, and the optimizer has finished executing the parameter update. The division method (number, size) of multiple sample batches can be dynamically adjusted according to the model structure and hardware configuration. The triggering conditions for data loading / unloading can be further implemented using auxiliary mechanisms such as timers and memory monitoring.

[0056] In one alternative implementation, the first device can operate as a single device, and the model update task processes multiple sample batches through gradient accumulation. Specifically, the forward and backward propagation processes for the next sample batch are executed after the forward and backward propagation processes for the current sample batch are completed. Correspondingly, the data loading task starts before the backward propagation process for the last sample batch begins. Gradient accumulation refers to updating parameters uniformly after accumulating gradients from multiple sample batches, equivalent to large-batch training. By processing multiple sample batches serially and updating uniformly after gradient accumulation, memory usage can be significantly reduced. Furthermore, determining the loading timing before the backward propagation of the last sample batch hides communication latency.

[0057] In another alternative implementation, the first device is any device in a pipelined parallel system. Pipeline parallelism refers to splitting the model layer by layer across multiple devices, allowing these devices to process the data in parallel in a pipelined manner. Correspondingly, the model update task executes the backpropagation process for multiple sample batches after the forward propagation process of multiple sample batches has completed. In this approach, the data loading task starts before the backpropagation process of the first sample batch among the multiple sample batches begins. In related technologies, the lack of pre-loading of state during forward propagation leads to devices idling and waiting at the start of backpropagation, resulting in high overhead for state synchronization between multiple devices. This approach loads data immediately after all sample batches have completed forward propagation, avoiding pipeline blockage and improving execution efficiency.

[0058] Therefore, the above method allows for the selection of a more suitable time to unload the optimizer state and implements asynchronous unloading. For example, assuming the first device is a GPU and the second device is a CPU, the first state data is the optimizer state data generated in the previous iteration, and the second state data is the optimizer state data generated in the current iteration. In a single-card scenario, multiple micro-batches can be used for gradient accumulation. Correspondingly, when the forward propagation of the last micro-batch ends and the backward propagation begins, the optimizer state is loaded onto the GPU. After loading, parameters are updated through the main computation stream (default stream), and then the optimizer state is randomly unloaded onto the CPU. For example, the main computation stream processes the forward and backward propagation processes of n batches in the following order: forward propagation of batch 1 - backward propagation of batch 1 - forward propagation of batch 2 - backward propagation of batch 2. Forward propagation of batch n - backward propagation of batch n. Accordingly, after the forward propagation of batch n ends and before the backward propagation of batch n begins, stream1 performs the loading process of the first state data, enabling the default stream to perform parameter update operations based on the loaded first state data and obtain the second state data based on the update results. Stream2 performs the unloading operation of the second state data. Furthermore, during the unloading operation of the second state data, the forward propagation of batch 1 in the next iteration round is simultaneously performed.

[0059] Furthermore, a similar approach is used in the PP pipeline parallel mode. In this mode, multiple devices process different parts of the model. Within each device, the following processing is performed via the default stream: first, multiple batches of forward propagation are processed, then multiple batches of backward propagation are processed, and finally, parameter updates are performed. Specifically, for any device in the pipeline parallel mode, when the forward propagation of the last microbatch of that device ends and the backward propagation begins, the optimizer state is loaded via stream1. And, immediately after the parameter update, the optimizer state is unloaded via stream2. It's worth noting that two new streams, stream1 and stream2, are needed for loading and unloading. These two streams can be custom CUDA streams used to asynchronously execute optimizer state loading (H2D) and unloading (D2H), and they are used to execute in parallel with the default stream. The communication process between H2D and D2H is hidden through time overlap. H2D (Host-to-Device) is used to transfer data from CPU memory to GPU memory, such as loading optimizer status. D2H (Device-to-Host) is used to transfer data from GPU memory to CPU memory, such as unloading optimizer status.

[0060] II. Further optimization by dividing the data into subgroups:

[0061] In one optional implementation, to further improve the communication overlap time, the first state data and the second state data can be pre-divided into multiple sub-data groups, each sub-data group corresponding to multiple parameters to be updated in the parameter update operation. Accordingly, when the first state data stored in the second device is loaded to the first device via a data loading task, the multiple sub-data groups in the first state data are loaded to the first device sequentially according to a first preset order. Furthermore, when the second state data stored in the first device is unloaded to the second device via a data unloading task, the multiple sub-data groups in the second state data are unloaded to the second device sequentially according to a second preset order. The multiple sub-data groups in the second state data correspond to the multiple sub-data groups in the first state data.

[0062] In this context, a sub-data group refers to a logical grouping of model parameters or optimizer states, which can be divided by layer, parameter quantity, or function. The sub-data groups for the second state data and the first state data are typically divided using the same grouping rules. Therefore, the number of sub-data groups for the second state data and the first state data is usually the same, and each group corresponds to the others. A first preset order represents the loading priority strategy; for example, it can be flexibly set according to parameter size, layer order, or memory requirements. Similarly, a second preset order represents the unloading priority strategy, which can also be flexibly set according to parameter size, layer order, or memory requirements. By dividing the data into sub-data groups, sub-data groups can be loaded / unloaded on demand, avoiding peak memory usage of the full state data. Furthermore, the loading, updating, and unloading processes between multiple sub-data groups can be executed in parallel, maximizing hardware utilization and minimizing hidden communication latency. Moreover, parameter updates can be performed partially on a parameter group basis, improving training flexibility.

[0063] In one optional implementation, the process of the data loading task loading the (i+1)th sub-data group in the first state data overlaps with the process of the model update task updating parameters based on the ith sub-data group in the first state data; and / or, the process of the data loading task loading the (i+1)th sub-data group in the first state data overlaps with the process of the data unloading task unloading the ith sub-data group in the second state data; where i is a natural number. "Overlapping" means that the two tasks are executed in parallel through different hardware execution units (such as CUDA streams), and their execution times partially or completely overlap. Therefore, overlap includes partial or complete overlap, which can be selected by those skilled in the art according to actual needs. Through the above method, the loading, unloading, and updating processes corresponding to different sub-data groups can be executed in parallel, thereby maximizing hardware resource utilization. Furthermore, the video memory released after the ith group is unloaded can be immediately used for loading the (i+1)th group, thereby achieving dynamic reuse of video memory resources.

[0064] In practice, overlapping loading and updating can be achieved in various ways: for example, when the update of the i-th group begins, the loading of the (i+1)-th group can be triggered immediately. Alternatively, the loading start timing can be dynamically adjusted based on GPU computing power (e.g., loading begins when 50% of the update is complete). This application does not limit the specific details of the overlap. Similarly, overlapping loading and unloading can be achieved in various ways. For example, assuming that in single-card training, the 12 Transformer layers are divided into 12 sub-groups, each containing one layer of parameters and associated optimizer states, processing between multiple sub-groups can be achieved in the following ways:

[0065] Time T1: Group1 is loaded via stream1, thus updating group1 via the default stream;

[0066] At time T2: group2 is loaded via stream1, and at the same time, group1 is unloaded via stream2 (the loading of group2 coincides with the unloading of group1).

[0067] At time T3: group2 is updated via the default stream, while group3 is loaded via stream1 (the update of group2 and the loading of group3 coincide).

[0068] In summary, this application does not limit the specific overlapping methods of loading, updating, and unloading processes among multiple sub-data groups, as long as it can solve the serial bottleneck of computation and communication in traditional training and improve training efficiency.

[0069] In one optional implementation, the parameter update result of the current iteration includes the update result of the first state data. Correspondingly, the i-th sub-data group in the second state data is determined based on the update result of the i-th sub-data group in the first state data; and after the model update task updates according to the i-th sub-data group in the first state data, the i-th sub-data group in the second state data is unloaded through a data unloading task. Thus, the first state data (also called historical state) typically includes information such as the optimizer state updated in the previous iteration, and is usually stored in the second device, needing to be loaded onto the first device for use in the current iteration. The second state data (also called current state) typically includes information such as the latest optimizer state generated in the current iteration, and usually needs to be unloaded onto the second device for loading in the next iteration. Both update and unloading operations can be implemented on a sub-data group basis, thereby hiding communication latency. Specifically, the process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the model update task updating parameters based on the (i+1)-th sub-data group in the first state data; and / or, the process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the data loading task loading the (i+1)-th sub-data group in the first state data.

[0070] For example, suppose the first state data corresponding to iteration 1 is stored in the CPU (second device), containing the optimizer state generated in iteration 1. In the training process of iteration 2, firstly, the aforementioned first state data is loaded to the GPU (first device), a loading process overlapping with the backpropagation process of iteration 2; then, parameters are updated to generate the second state data corresponding to iteration 2; finally, the second state data is unloaded to the CPU, an unloading process overlapping with the forward propagation process of iteration 3. The first state data is further divided into N sub-data groups, and correspondingly, the loading, updating, and unloading processes are all processed on a sub-data group basis. For example, after the first sub-data group in the second state data is loaded, the corresponding update operation can be performed to obtain the first sub-data group in the first state data; during the update process, the loading process of the second sub-data group in the second state data can be performed simultaneously. Then, the unloading operation of the first sub-data group in the first state data is performed. During the unloading process, the corresponding update operation can be performed simultaneously based on the second sub-data group in the second state data to obtain the second sub-data group in the first state data, and so on. In this way, the asynchronous parallel processing of loading, updating, and unloading of multiple sub-data groups can be achieved, reducing communication latency.

[0071] Therefore, the above method can further optimize memory and communication through subgroups. For example, to further optimize memory and communication, the parameter groups (param_groups) corresponding to the parameters (params) that need to be unloaded in the optimizer state can be split into several sub-param_groups. Correspondingly, after the backpropagation of the last micro-batch is completed, the optimizer state may not be fully loaded, and the parameter update process of the optimizer will be blocked. With several sub-param_groups, as long as the optimizer state corresponding to any parameter in a sub-param_group has been fully loaded, parameter updates can be performed for that sub-param_group. This does not affect the parameter updates of the optimizer states corresponding to other sub-param_groups.

[0072] In addition to the advantages of a single `param_group`, multiple `sub_param_groups` can alleviate memory pressure: after the parameters of the previous `sub_param_group` are updated, the state is immediately unloaded. Memory release is significantly accelerated, and the freed-up memory can be directly used to load the optimizer state of the next `sub_param_group`. Furthermore, the loading process of the optimizer state of the next `sub_param_group` and the unloading process of the optimizer state of the previous `sub_param_group` can overlap. This not only alleviates memory pressure but also improves bandwidth utilization.

[0073] III. Optimizations for loading and / or unloading order:

[0074] In one optional implementation, to further improve the space utilization of the first device, the first preset order is an order in which the data volume of multiple sub-data groups in the first state data increases sequentially, and the second preset order is an order in which the data volume of multiple sub-data groups in the second state data decreases sequentially. Here, data volume refers to the storage size of parameters in a sub-data group, which can be represented by units such as bytes, and can be determined by the product of the number of parameters (elements) in the sub-data group and the number of bytes in a single element. Since the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration, and during backpropagation, gradients are calculated in the order from the output layer to the input layer, and used activation values ​​are released layer by layer, resulting in a gradual decrease in GPU memory usage, based on the characteristic of gradually decreasing GPU memory usage during backpropagation, loading smaller sub-data groups first can initiate the update process of smaller sub-data groups as much as possible in the early stages of backpropagation when GPU memory usage is high, thereby making full use of the hardware's spare capacity. Furthermore, in the later stages of backpropagation when GPU memory usage is low, loading larger sub-data groups will not cause space shortage problems. Similarly, since the execution process of the data unloading task overlaps with the forward propagation process of the model update task for the next iteration, and during forward propagation, the input activation values ​​of each layer need to be cached for gradient calculation during backpropagation, leading to a gradual increase in GPU memory usage, based on the characteristic of gradually increasing GPU memory usage during forward propagation, unloading larger sub-data groups first can quickly release GPU memory resources in the early stages of forward propagation. Therefore, loading starts with small data groups and gradually expands GPU memory usage to avoid a sudden surge in GPU memory pressure; unloading starts with large data groups to quickly release GPU memory and prioritize alleviating the GPU memory bottleneck.

[0075] Therefore, the above method can select a reasonable order for unloading and loading optimizer states. For example, during backpropagation, the activation values ​​saved during forward propagation are gradually released, the GPU memory pressure gradually decreases, and the available GPU memory gradually increases. Therefore, by sorting the optimizer states to be loaded from smallest to largest, smaller optimizer states are loaded first, allowing loading to begin as quickly as possible. After the parameter update is complete, the optimizer state unloading process begins immediately on stream 2. At this time, the GPU memory occupied by forward propagation increases from smallest to largest; therefore, unloading is performed in descending order of optimizer states to reduce memory usage as quickly as possible, ensuring the smooth progress of the forward propagation process.

[0076] IV. Optimization through partial uninstallation mechanism:

[0077] In one optional implementation, when unloading the second state data stored in the first device to the second device via a data unloading task, it can be achieved as follows: Based on a preset unloading ratio, the second state data corresponding to the current iteration in the first device is divided into an unloaded data group and a retained data group. The unloaded data group is then unloaded to the second device via a data unloading task, so that the unloaded data group serves as the first state data stored in the second device in the next iteration. The amount of data in the unloaded data group matches the unloading ratio; the retained data group is stored in the first device to enable parameter updates in the next iteration. In this method, based on a preset unloading ratio (e.g., 30%), the second state data of the current iteration can be divided into two parts: one part to be unloaded to the second device (i.e., the unloaded data group), and the other part to be retained in the first device (i.e., the retained data group). Accordingly, the unloaded data group serves as the first state data stored in the second device in the next iteration, while the retained data group is directly used for parameter updates in the next iteration. This method can unload some state data as needed and retain key data in GPU memory to balance memory usage and computing efficiency. Since the retained data group directly participates in the next round of calculation, it can avoid repeated loading and reduce communication overhead.

[0078] The Offload Ratio refers to the percentage of second-state data to be offloaded out of the total state data, which can be statically preset or dynamically calculated. The Offload Data Group is a set of sub-data groups selected based on the Offload Ratio, which may contain low-priority parameters (such as shallow network weights) or optimizer states with low update frequency (such as Adam's v variance term). The Retained Data Group is a set of sub-data groups stored in GPU memory, which may contain high-priority parameters (such as classifier head weights) or optimizer states with high update frequency (such as Adam's m momentum term). In practice, the selection strategy for the Offload Data Group can be chosen based on factors such as the number of parameters and update frequency. For example, data groups can be sorted by the number of parameters, prioritizing the offloading of the largest sub-data group to quickly release GPU memory. Alternatively, data groups can be sorted by update frequency, prioritizing the offloading of parameters with slow gradient updates. Correspondingly, the Retained Data Group can be fully retained and directly participate in the next round of forward / backward computation without reloading. Or, to avoid parameter bias caused by long periods without updates, the Retained Data Group can be updated synchronously with the reloaded data group before the next round of forward propagation.

[0079] Traditional full unload / load schemes are prone to causing periodic idle memory (e.g., a full load is still required even when only some parameters need updating). Furthermore, repeatedly transmitting unmodified state data (e.g., Adam's v updates slowly in the early training phase) can lead to communication redundancy. Frequent loading / unloading of high-priority parameter groups can also increase critical path latency. The method described above, through dynamic proportional unloading and partial state retention, achieves coordinated optimization of memory, communication, and computing resources. It can unload non-critical data as needed, retain high-value parameters in resident memory, thereby avoiding repeated transmissions and improving hardware utilization.

[0080] Therefore, the above method provides an adjustable partial unloading mechanism (also known as a hybrid unloading mechanism). In some cases, the GPU memory shortage is not significant, and it is not necessary to unload the entire optimizer state. In this case, a partial unloading / hybrid unloading mechanism can be used: the user can set a scaling factor γ to control the proportion of optimizer state unloading. In practice, each trainable parameter param during training corresponds to a first moment (momentum term epx_avg) and a second moment (variance term epx_avg_sq) in the optimizer state. For example, assuming there are n trainable parameters, param 1 corresponds to epx_avg 1 and epx_avg_sq 1, param 2 corresponds to epx_avg 2 and epx_avg_sq 2, ..., param n corresponds to epx_avg n and epx_avg_sq n.

[0081] In one example, assuming the total number of trainable parameters `params` is A, since each trainable parameter corresponds to two optimizer state elements, the theoretical number of optimizer states elements to be unloaded should be [A]. ,the remaining It remains on the GPU. It's worth noting that since the optimizer state is typically organized as tensors, the parameters corresponding to a complete tensor should not be split; the number of elements unloaded should be greater than or equal to... The minimum value under the given conditions. Of course, besides splitting data groups based on tensors, data groups can also be split according to tensor blocks, layers, etc.

[0082] V. Further optimization of the partial uninstallation mechanism through an alternating uninstallation mechanism:

[0083] In one optional implementation, the second state data in the first device corresponding to the current iteration round includes: a first data group determined based on the update results of the retained data group from the previous iteration round, and a second data group determined based on the update results of the unloaded data group from the previous iteration round. The first data group is the update result generated by calculating the parameters (retained data group) retained in the GPU from the previous round in the current round, and the second data group is the update result generated by calculating the parameters (unloaded data group) unloaded to the CPU from the previous round and then reloaded in the current round. Accordingly, when dividing the second state data in the first device corresponding to the current iteration round into an unloaded data group and a retained data group according to a preset unloading ratio, the first data group can be designated as the unloaded data group in the current iteration round, and the second data group can be designated as the retained data group in the current iteration round. In this approach, by assigning the historical retained group (first data group) to the new unloaded group and the historical unloaded group (second data group) to the new retained group, the rotational update of state data is achieved. In this approach, the retained group can be directly reused, avoiding repeated loading.

[0084] The unloading ratio and grouping strategy can be implemented in several ways. For example, it can be achieved through a fixed rotation: when the unloading ratio is 50%, the historical reserved group (first data group) is used as the new unloaded data group, and the historical unloaded group (second data group) is used as the new reserved data group, ensuring that all parameters participate in memory-CPU rotation periodically. Alternatively, it can be implemented by combining dynamic selection: dynamically selecting the unloaded data group based on parameter importance (such as gradient magnitude), prioritizing the unloading of less important parameters.

[0085] The above method, through rotational unloading, allows the historical retention group to be directly reused, thereby reducing repeated loading and lowering the peak memory usage.

[0086] Therefore, the above method provides a dynamic hybrid unloading mechanism:

[0087] The scaling factor γ mentioned above is represented by a fraction. When fraction < 1, better optimization can be achieved by alternately and dynamically unloading the optimizer state. For example, all parameter groups can be divided: the parameter group where the optimizer state resides on the GPU is called the GPU parameter group, and the parameter group where the optimizer state resides on the CPU is called the CPU parameter group. During each update operation, the parameters of the GPU parameter group are updated first, and after the update, the parameters of that parameter group are directly unloaded. At this point, the GPU parameter group is transformed into the CPU parameter group in the next iteration. When the number of unloaded parameters reaches... Afterwards, the parameter groups are divided into GPU parameter groups. These GPU parameter groups are not unloaded after parameter updates; they are processed first in the next optimizer parameter update without needing to be reloaded. Therefore, by using the GPU parameter group in the current iteration as the CPU parameter group in the next iteration, and vice versa, a dynamic alternation of unloading and retaining data groups can be achieved, further improving model update efficiency.

[0088] In one optional implementation, before dividing the second state data corresponding to the current iteration in the first device into an unloaded data group and a retained data group according to a preset unloading ratio, the following operations are further performed: if it is detected that the remaining device space of the first device in the previous iteration is greater than a preset value, the unloading ratio corresponding to the current iteration is determined based on the comparison result between the remaining device space and the preset value. Here, the remaining device space is the available video memory capacity of the first device (GPU). The preset value (threshold) can be a lower limit of video memory set according to hardware conditions, used to trigger the adjustment of the unloading ratio. Therefore, before dividing the unloaded data group and the retained data group, the remaining video memory space of the first device in the previous iteration can be further detected, and the unloading ratio can be dynamically calculated: if the remaining video memory space is greater than the preset threshold, the unloading ratio for the current iteration is dynamically calculated based on the difference between the remaining space and the threshold, and the calculated unloading ratio is used to divide the second state data for the current iteration. Specifically, when video memory is sufficient, the unloading ratio can be reduced to retain more parameters in the GPU; when video memory is scarce, the unloading ratio can be increased to balance computational efficiency and resource constraints. This method can adaptively match the model size and hardware configuration.

[0089] VI. Improve optimization results by increasing the unloading domain:

[0090] In one optional implementation, the first state data stored in the second device includes: optimizer state data determined based on the parameter update result of the previous iteration, and / or model parameter data of first precision; wherein the model parameter data of first precision is used to implement the parameter update operation of the current iteration, and the forward propagation process and backward propagation process of the current iteration are determined by the model parameter data of second precision, wherein the first precision is greater than the second precision; the second state data in the first device includes: optimizer state data determined based on the parameter update result of the current iteration, and / or model parameter data of first precision.

[0091] Therefore, the first state data stored in the second device includes the following two parts: (1) optimizer state data: intermediate optimizer variables (such as Adam's momentum m and variance v) determined based on the parameter update results of the previous iteration; (2) first-precision model parameter data: high-precision parameter copies (such as FP32 master parameters), used for parameter update operations in the current iteration. Similarly, the second state data stored in the first device also includes the following two parts: (1) optimizer state data: the latest optimizer state generated based on the parameter update results of the current iteration; (2) first-precision model parameter data: high-precision parameter copies updated in the current iteration. Among them, the first precision (high precision) such as FP32 is used for parameter updates to ensure numerical stability; the second precision (low precision) such as FP16 / BF16 is used for forward propagation and backward propagation to improve computational efficiency.

[0092] Therefore, in mixed-precision mode, the main parameter (main_param) can also be included in the unloading scope. In mixed-precision scenarios, the model's forward and backward propagation processes are implemented based on low-precision parameters, while the parameter update process is implemented based on high-precision parameters. In this case, the model needs to maintain a high-precision copy, called main_param. Correspondingly, this main_param can also be added to the unloading scope in the unloading strategy. This makes the unloading domain larger, and the control over GPU memory also becomes larger.

[0093] To facilitate understanding, an example is provided below to illustrate the implementation process of the model update method in this application. This example involves interdisciplinary fields such as computer system architecture, machine learning acceleration, and heterogeneous computing resource management. It provides an optimization scheme based on dynamic task offloading and hardware collaboration, which can offload the optimizer state (such as the first and second moments of gradients) during machine learning training from dedicated hardware acceleration units (such as GPUs, NPUs, or dedicated AI chips) to traditional hardware (such as CPU architecture). This example mainly addresses issues such as GPU memory bottlenecks, memory bandwidth limitations, and uneven utilization of heterogeneous computing resources caused by optimizer states in deep learning training and distributed model fine-tuning scenarios. It also supports emerging application scenarios such as edge computing, federated learning, and mixed-precision training.

[0094] In related technologies, with the explosive growth of deep learning model sizes (such as Transformer, LLM, etc.), GPU memory has become a key bottleneck for training. When GPUs are plentiful, data parallelism, tensor parallelism, and model parallelism can be used to optimize the memory on a single card. When the number of GPUs is insufficient, memory optimization techniques, such as reducing the memory footprint of model parameters, intermediate activation values, and gradients, can significantly improve training efficiency and model scalability.

[0095] When GPUs are limited, common strategies for conserving GPU memory include recompute, activation offload, CPU optimizer, and optimizer offload. Recompute involves selectively recompiling some intermediate activation values ​​during backpropagation, rather than storing them all in the forward propagation. Activation offload transfers data from GPU memory to CPU memory: intermediate activation values ​​are temporarily unloaded into CPU memory and loaded back into the GPU when needed. The CPU optimizer stores the main parameters and optimizer state in CPU memory, and also transmits gradients to the CPU after calculation. Parameter updates are performed on the CPU and then transmitted back to the GPU. All optimization-related operations are performed on the CPU, with the GPU primarily responsible for forward / backward computation. Optimizer offload resides optimizer parameters (such as Adam's momentum and variance) in CPU memory, not GPU memory, and loads them into the GPU only when needed. For example, only the optimizer state (such as Adam's m / v) can be unloaded to the CPU, while the main parameters and gradients remain on the GPU.

[0096] However, all of the above solutions have certain drawbacks and shortcomings:

[0097] Recomputation methods require recalculating intermediate activation values ​​during backpropagation, increasing computation time by 20%-50%. Activation offloading effectively saves GPU memory, but in all scenarios, it requires frequent CPU-GPU data transfers, which can cause severe memory fragmentation in many situations (especially in pipelined parallel training), reducing memory utilization and significantly increasing communication overhead. For example, after forward propagation, unnecessary activation values ​​need to be asynchronously copied from GPU memory to CPU memory; when needed for backpropagation, a prefetching mechanism loads the corresponding activation values ​​back to the GPU in advance. CPU Optimizers require optimizer step operations on the CPU, resulting in significant performance degradation. For example, parameters and optimizer states reside on the CPU, and optimizer updates are performed by the CPU. Optimizer stateoffload methods block optimizer step parameter updates during offloading and reloading, increasing update latency. The optimizer step represents the method function and is used to update model parameters based on the currently calculated gradients. Meanwhile, since the optimizer state still needs to be loaded onto the GPU when updating parameters, its effect on reducing video memory may not be obvious.

[0098] To address the aforementioned issues, this example proposes a model update method that effectively and controllably conserves GPU memory while minimizing performance loss, even under conditions of limited GPU memory, thereby maximizing GPU computational efficiency and memory utilization. This example can be improved upon the optimizer state offload scheme: in many memory-constrained scenarios, gradient accumulation or pipeline parallelism is employed. In gradient accumulation, each step's mini-batch is split into multiple micro-batches during training. Each micro-batch only performs forward and backward operations without parameter updates. Parameter updates are performed only after accumulating num-micro-batch times. Therefore, throughout the entire step's lifecycle, the optimizer's state remains unused for a long period. Thus, offloading it is theoretically the optimal choice, as it is not as frequently unloaded and loaded as activation values.

[0099] Figure 2 A schematic flowchart of a model update method provided in another embodiment of this disclosure is shown. Figure 2As shown, the model update method includes the following steps:

[0100] Step S201: Start the training task and perform initialization.

[0101] In a typical multi-GPU pipelined parallel distributed training task, the micro-batch size can be set to 1, the num-micro-batch size to 32, and the startup script set to enable optimizer offload with a fraction of 1 (fully offloaded). It's important to note that during the initial training phase, the fraction should be set to 1 initially, and then adjusted as needed if GPUs become available.

[0102] Assuming the optimizer has one param_group, two streams are pre-established: stream1 for H2D operations and stream2 for D2H operations. Additionally, default operations are performed via the default stream.

[0103] Step S202: In the first iteration, after the parameters are updated, the state data is unloaded to the CPU.

[0104] During GPU training, forward propagation is performed first, followed by back propagation, and finally parameter updates. Forward propagation calculates model predictions and the loss function, with its memory usage gradually increasing. Back propagation calculates the gradient of the loss function with respect to the model parameters, with its memory usage gradually decreasing. During parameter updates, the model parameters are updated based on the gradients. For example, in the first iteration, after the model update, the state data is unloaded from the GPU to the CPU.

[0105] It's important to note that during the first step, the optimizer state is not yet initialized and does not need to be loaded from the CPU. After the first step, the optimizer state is randomly unloaded via stream2. During this unloading process, the optimizer states of different tensors are sorted, with larger optimizer tensors being unloaded first.

[0106] Step S203: During the backpropagation of the second iteration of the model update task, the state data stored in the CPU in the first iteration is loaded to the GPU through the data loading task.

[0107] The second iteration also includes forward propagation, backward propagation, and parameter update. In the multi-batch scenario, when the forward propagation of the last micro-batch finishes and backward propagation of the loss begins, the optimizer is loaded via stream1 to transfer the state data stored in the CPU from the first iteration to the GPU. It's important to note that the GPU memory used to store activation values ​​is released gradually; therefore, the optimizer states to be loaded can be sorted from smallest to largest, prioritizing the loading of smaller optimizer states.

[0108] Step S204: Based on the state data stored in the CPU in the first iteration, perform the parameter update operation in the second iteration, and obtain the state data corresponding to the second iteration based on the update result.

[0109] Step S205: During the forward propagation of the third iteration, the state data corresponding to the second iteration is unloaded to the GPU. Specifically, the optimizer state is unloaded via stream2.

[0110] Figure 2 Only the first three iterations are shown in the diagram. The basic training version can be achieved by repeating the above steps in subsequent iterations. In summary, before the first device performs the parameter update operation in the i-th iteration via the model update task, the first state data stored in the second device in the (i-1)-th iteration is loaded into the first device via the data loading task. Furthermore, after the model update task performs the parameter update operation in the i-th iteration, the second state data stored in the first device in the i-th iteration is unloaded into the second device via the data unloading task, for loading in the (i+1)-th iteration.

[0111] Additionally, it should be noted that during the execution of any iteration, the following judgment can also be performed: if there is a large amount of available video memory remaining, the unloading ratio fraction can be adjusted to maximize the utilization of available video memory.

[0112] For example, when fraction < 1, the optimizer will split optimizer param_groups into two parameter groups. The first parameter group is not unloaded, i.e., it is the data group to be retained; the second parameter group needs to be unloaded, i.e., it is the data group to be unloaded. When updating, the parameters of the first parameter group can be updated first.

[0113] In many cases, after the reverse process of the last micro-batch is completed, the loading process of the optimizer state may not have finished, which can cause the optimizer to be blocked. In this case, the param_groups that need to be unloaded are further split into multiple sub_param_groups and the parameters are updated one by one. After the parameters of each param_group are updated, the unloading is performed immediately, thereby overlapping the H2D and D2H communication operations in stream1 and stream2.

[0114] Optionally, during the execution of any iteration, the current communication overlap ratio can be further determined. If there are still cases where communication is not overlapped, i.e., the communication overlap between different processes is insufficient, and the unloading ratio is <1, a dynamic alternating unloading mechanism can be used for further optimization: when performing a step, the parameters of the unloaded parts of the param_group are updated first, and after the update, this part becomes unloadable and is immediately unloaded, thereby realizing the dynamic rotation between unloaded data groups and retained data groups. When the number of unloaded parameters is sufficient, the remaining param_groups do not need to be unloaded and become retained data groups.

[0115] In summary, this example provides a solution to the GPU memory limitation problem, with the core focus on minimizing the impact on training performance. On resource-constrained devices (such as single-GPU or multi-GPU clusters with insufficient memory), this method can overcome hardware limitations and democratize the training of models with hundreds of billions or even trillions of parameters. This method is compatible with mixed-precision training (FP16 / FP32), gradient accumulation, and ZeRO (zero redundancy optimizer), forming a more efficient training solution.

[0116] In summary, the key point of this example is that the specific process of optimizer offload has been optimized in a fine-grained manner, and a number of improvement measures have been proposed, including unloading timing, unloading order, hybrid unloading strategy, parameter group splitting, and dynamic unloading strategy. Among them, it has at least the following advantages: (1) An asynchronous unloading strategy is adopted to unload and load the optimizer state at the most appropriate time, reducing the impact on the default stream; (2) During unloading and loading, the size of the optimizer state corresponding to each parameter is sorted, and unloading and loading are carried out in a planned manner; (3) The unloading ratio can be adjusted to adapt to the memory gap and maximize the memory utilization; (4) The sub_param_groups mechanism is used to update parameters to further save memory usage and optimize communication overlap; (5) The dynamic offload mechanism alternates between different param_groups to optimize performance to the extreme.

[0117] In addition, another embodiment of this disclosure provides a data processing method, which includes: inputting data to be processed into a target data processing model to obtain a data processing result corresponding to the data to be processed; wherein the data to be processed can be any one of text to be processed, speech to be processed, image to be processed, or video to be processed; wherein the target data processing model can be obtained according to the model update method mentioned above.

[0118] The model update method disclosed herein can be flexibly applied to various types of data, and the relevant parameters in the model update method can also be adapted according to the task type. For example, in the model update method, the parameter configuration can be dynamically adjusted according to the task type (text, speech, image, video, etc.), data characteristics (sequence length, spatial dimension, temporal features), and hardware resources (video memory, computing power).

[0119] Figure 3 This is a block diagram of a model update apparatus provided in an embodiment of this disclosure. (Refer to...) Figure 3 This disclosure provides a model update apparatus, which includes:

[0120] The loading module 31 is adapted to load first state data stored in the second device into the first device through a data loading task; wherein the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially execute forward propagation, backpropagation and parameter update operations in each iteration round.

[0121] Update module 32 is adapted to perform parameter update operation for the current iteration round through the model update task based on the first state data;

[0122] The unloading module 33 is adapted to unload the second state data stored in the first device to the second device through a data unloading task; wherein the execution process of the data unloading task coincides with the forward propagation process of the model update task executing the next iteration round; and the second state data is determined according to the parameter update result of the current iteration round and is used to execute the parameter update operation of the next iteration round.

[0123] In one alternative implementation, the model update device is implemented based on multiple sample batches;

[0124] The loading module is also adapted to: initiate the data loading task after the forward propagation process of the last sample batch in the plurality of sample batches has ended;

[0125] Furthermore, the unloading module is also adapted to initiate the data unloading task after the parameter update operations corresponding to the multiple sample batches have been completed.

[0126] In one optional implementation, the first device operates in a single-device state, and the model update task processes the multiple sample batches through gradient accumulation; wherein, after the forward and backward propagation processes of the current sample batch are completed, the forward and backward propagation processes of the next sample batch are executed; then the data loading task starts before the backward propagation process of the last sample batch begins; or,

[0127] The first device is any device in the pipelined parallel system, and the model update task executes the backpropagation process of the multiple sample batches after the forward propagation process of the multiple sample batches is completed; then the data loading task starts before the backpropagation process of the first sample batch in the multiple sample batches begins.

[0128] In one optional implementation, the first state data and the second state data are pre-divided into multiple sub-data groups, and the multiple sub-data groups correspond to multiple parameters to be updated in the parameter update operation;

[0129] The loading module is specifically adapted to: sequentially load multiple sub-data groups in the first state data into the first device according to a first preset order;

[0130] The unloading module is specifically adapted to: sequentially unload multiple sub-data groups in the second status data to the second device according to a second preset order;

[0131] In this context, multiple sub-data groups in the second state data correspond to multiple sub-data groups in the first state data.

[0132] In one optional implementation, the process of the data loading task loading the (i+1)th sub-data group from the first state data overlaps with the process of the model update task updating the first state data based on the i-th sub-data group; and / or,

[0133] The process of the data loading task loading the (i+1)th sub-data group in the first state data overlaps with the process of the data unloading task unloading the ith sub-data group in the second state data; where i is a natural number.

[0134] In one optional implementation, the parameter update result of the current iteration includes the update result of the first state data, and the i-th sub-data group in the second state data is determined based on the update result of the i-th sub-data group in the first state data;

[0135] Furthermore, after the model update task updates the data according to the i-th sub-data group in the first state data, the i-th sub-data group in the second state data is unloaded by the data unloading task.

[0136] The process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the model update task updating the model based on the (i+1)-th sub-data group in the first state data; and / or, the process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the data loading task loading the (i+1)-th sub-data group in the first state data.

[0137] In one optional implementation, the first preset order is the order in which the data volume of the multiple sub-data groups in the first state data increases sequentially, and the second preset order is the order in which the data volume of the multiple sub-data groups in the second state data decreases sequentially.

[0138] In one alternative implementation, the unloading module is specifically adapted to:

[0139] According to the preset unloading ratio, the second state data in the first device corresponding to the current iteration round is divided into an unloading data group and a retention data group. The unloading data group is unloaded to the second device through a data unloading task, so that the unloading data group is used as the first state data stored in the second device in the next iteration round.

[0140] The amount of data in the unloaded data group is matched with the unload ratio; the retained data group is stored in the first device to enable parameter update operations in the next iteration.

[0141] In one optional implementation, the second state data in the first device corresponding to the current iteration round includes: a first data group determined based on the update result of the retained data group in the previous iteration round, and a second data group determined based on the update result of the unloaded data group in the previous iteration round;

[0142] The unloading module is specifically adapted to: determine the first data group as the unloaded data group in the current iteration round according to a preset unloading ratio, and determine the second data group as the retained data group in the current iteration round.

[0143] In one alternative implementation, the unloading module is further adapted to:

[0144] If it is detected that the remaining space of the first device in the previous iteration round is greater than a preset value, the unloading ratio corresponding to the current iteration round is determined based on the comparison result between the remaining space of the device and the preset value.

[0145] In one optional implementation, the first state data stored in the second device includes: optimizer state data determined based on the parameter update result of the previous iteration, and / or model parameter data of a first precision; wherein the model parameter data of the first precision is used to implement the parameter update operation of the current iteration, and the forward propagation process and backward propagation process of the current iteration are determined by the model parameter data of a second precision, wherein the first precision is greater than the second precision;

[0146] The second state data in the first device includes: optimizer state data determined based on the parameter update results of the current iteration round, and / or model parameter data of the first precision.

[0147] Another embodiment of this disclosure also provides a data processing apparatus, including:

[0148] The data processing module is adapted to input the data to be processed into the target data processing model to obtain the data processing result corresponding to the data to be processed; wherein, the data to be processed is any one of the following: text to be processed, speech to be processed, image to be processed, and video to be processed;

[0149] The target data processing model is obtained based on the aforementioned model update device.

[0150] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0151] Reference Figure 4 This disclosure provides an electronic device, which includes: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by the at least one processor 401, and the one or more computer programs are executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described method.

[0152] In some embodiments, the processing device can be a neuromorphic chip. Since neuromorphic chips can employ vectorized computation and require external memory, such as Double Data Rate (DDR) synchronous dynamic random access memory, to load parameters like weights of the neural network model, the batch processing method used in this disclosure provides higher computational efficiency.

[0153] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program implements the above-described method when executed by a processor / processor core. The computer-readable storage medium may be volatile or non-volatile.

[0154] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0155] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0156] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0157] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0158] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0159] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0160] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0161] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0162] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0164] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model update method, characterized in that, The method is applied to a first device, and the method includes: The data loading task loads the first state data stored in the second device into the first device; wherein the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially execute forward propagation, backpropagation and parameter update operations in each iteration round; wherein the first state data is determined according to the parameter update result of the previous iteration round and is used to execute the parameter update operation in the current iteration round; Based on the first state data, the parameter update operation for the current iteration is performed through the model update task; The second state data stored in the first device is unloaded to the second device via a data unloading task; wherein the execution process of the data unloading task overlaps with the forward propagation process of the model update task for the next iteration round; and the second state data is determined based on the parameter update result of the current iteration round and is used to perform the parameter update operation for the next iteration round; wherein the first device is a GPU device or a video memory device, the second device is a CPU device or a memory device, and the first state data and the second state data include optimizer state data; wherein the data loading task and the data unloading task are independent of the model update task, and the data loading task and the data unloading task are executed asynchronously and in parallel with the model update task.

2. The method according to claim 1, characterized in that, The model update method is implemented based on multiple sample batches; Before loading the first state data stored in the second device to the first device through the data loading task, the method further includes: starting the data loading task after the forward propagation process of the last sample batch in the plurality of sample batches has ended; Furthermore, before unloading the second state data stored in the first device to the second device through the data unloading task, the method further includes: starting the data unloading task after the parameter update operations corresponding to the multiple sample batches are completed.

3. The method according to claim 2, characterized in that, The first device operates in a single-device state, and the model update task processes the multiple sample batches through gradient accumulation; wherein, after the forward and backward propagation processes of the current sample batch are completed, the forward and backward propagation processes of the next sample batch are executed; then the data loading task starts before the backward propagation process of the last sample batch begins; or, The first device is any device in the pipelined parallel system, and the model update task executes the backpropagation process of the multiple sample batches after the forward propagation process of the multiple sample batches is completed; then the data loading task starts before the backpropagation process of the first sample batch in the multiple sample batches begins.

4. The method according to claim 1, characterized in that, The method further includes: pre-dividing the first state data and the second state data into multiple sub-data groups, wherein the multiple sub-data groups correspond to multiple parameters to be updated in the parameter update operation; The step of loading the first state data stored in the second device to the first device through a data loading task includes: loading multiple sub-data groups in the first state data to the first device in a first preset order; The step of unloading the second state data stored in the first device to the second device through the data unloading task includes: unloading multiple sub-data groups in the second state data to the second device in a second preset order; In this context, multiple sub-data groups in the second state data correspond to multiple sub-data groups in the first state data.

5. The method according to claim 4, characterized in that, The process of the data loading task loading the (i+1)th sub-data group from the first state data overlaps with the process of the model update task updating parameters based on the ith sub-data group from the first state data; and / or, The process of the data loading task loading the (i+1)th sub-data group in the first state data overlaps with the process of the data unloading task unloading the ith sub-data group in the second state data; where i is a natural number.

6. The method according to claim 4, characterized in that, The parameter update result of the current iteration includes the update result of the first state data, and the i-th sub-data group in the second state data is determined according to the update result of the i-th sub-data group in the first state data; Furthermore, after the model update task updates the parameters according to the i-th sub-data group in the first state data, the i-th sub-data group in the second state data is unloaded by the data unloading task. The process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the model update task updating parameters based on the (i+1)-th sub-data group in the first state data; and / or, the process of the data unloading task unloading the i-th sub-data group in the second state data overlaps with the process of the data loading task loading the (i+1)-th sub-data group in the first state data.

7. The method according to any one of claims 4-6, characterized in that, The first preset order is the order in which the data volume of multiple sub-data groups in the first state data increases sequentially, and the second preset order is the order in which the data volume of multiple sub-data groups in the second state data decreases sequentially.

8. The method according to claim 1, characterized in that, The step of unloading the second state data stored in the first device to the second device through the data unloading task includes: According to the preset unloading ratio, the second state data in the first device corresponding to the current iteration round is divided into an unloading data group and a retention data group. The unloading data group is unloaded to the second device through a data unloading task, so that the unloading data group is used as the first state data stored in the second device in the next iteration round. The amount of data in the unloaded data group is matched with the unload ratio; the retained data group is stored in the first device to enable parameter update operations in the next iteration.

9. The method according to claim 8, characterized in that, The second state data in the first device corresponding to the current iteration round includes: a first data group determined based on the update result of the retained data group in the previous iteration round, and a second data group determined based on the update result of the unloaded data group in the previous iteration round; The step of dividing the second state data in the first device corresponding to the current iteration round into an unloading data group and a retained data group according to the preset unloading ratio includes: Based on the preset unloading ratio, the first data group is determined as the unloaded data group in the current iteration round, and the second data group is determined as the retained data group in the current iteration round.

10. The method according to claim 8 or 9, characterized in that, Before dividing the second state data in the first device corresponding to the current iteration into an unloading data group and a retained data group according to a preset unloading ratio, the method further includes: If it is detected that the remaining space of the first device in the previous iteration round is greater than a preset value, the unloading ratio corresponding to the current iteration round is determined based on the comparison result between the remaining space of the device and the preset value.

11. The method according to any one of claims 1-6, characterized in that, The first state data stored in the second device includes: optimizer state data determined based on the parameter update result of the previous iteration round, and / or model parameter data of first precision; wherein, the model parameter data of first precision is used to implement the parameter update operation of the current iteration round, and the forward propagation process and backward propagation process of the current iteration round are determined by the model parameter data of second precision, wherein the first precision is greater than the second precision; The second state data in the first device includes: optimizer state data determined based on the parameter update results of the current iteration round, and / or model parameter data of the first precision.

12. A data processing method, characterized in that, include: The data to be processed is input into the target data processing model to obtain the data processing result corresponding to the data to be processed; wherein, the data to be processed is any one of the following: text to be processed, speech to be processed, image to be processed, and video to be processed; The target data processing model is obtained based on the model update method described in any one of claims 1 to 11.

13. A model update device, characterized in that, The device is applied to the first apparatus, the device comprising: A loading module is adapted to load first state data stored in a second device into the first device via a data loading task; wherein the execution process of the data loading task overlaps with the backpropagation process of the model update task in the current iteration round; the model update task is used to sequentially execute forward propagation, backpropagation, and parameter update operations in each iteration round; wherein the first state data is determined based on the parameter update result of the previous iteration round and is used to execute the parameter update operation in the current iteration round; The update module is adapted to perform the parameter update operation of the current iteration round through the model update task based on the first state data; An unloading module is adapted to unload second state data stored in the first device to the second device via a data unloading task; wherein the execution process of the data unloading task overlaps with the forward propagation process of the model update task executing the next iteration round; and the second state data is determined according to the parameter update result of the current iteration round and is used to execute the parameter update operation of the next iteration round; wherein the first device is a GPU device or a video memory device, the second device is a CPU device or a memory device, and the first state data and the second state data include optimizer state data; wherein the data loading task and the data unloading task are independent of the model update task, and the data loading task and the data unloading task are executed asynchronously and in parallel with the model update task.

14. A data processing apparatus, characterized in that, include: The data processing module is adapted to input the data to be processed into the target data processing model to obtain the data processing result corresponding to the data to be processed; wherein, the data to be processed is any one of the following: text to be processed, speech to be processed, image to be processed, and video to be processed; The target data processing model is obtained based on the model update device in claim 13.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-12.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-12.

17. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that, When the computer-readable code is run in an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 12.

Images