Data processing method and device, equipment and storage medium
By performing model training on the GPU node during the intelligent model training process, determining whether the intermediate data meets the preset conditions, and storing the data to the CPU node when the conditions are met, the problem of waste of computing power resources caused by interrupts in intelligent model training is solved, and the continuity and efficiency of the training process are achieved.
Patent Information
- Application Number
- CN202411688808.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-18
AI Technical Summary
During the training of intelligent models, due to factors such as huge resources, complex data processing and long training time, we are prone to face the problem of training interruptions, resulting in wasting of computing resources.
Model training is performed in the GPU node to determine whether the intermediate data meets the preset conditions, and store the intermediate data and parameter gradients to the CPU node when the conditions are met. When the model training is interrupted, context recovery is completed based on the training phase and the prediction strategy.
Through this method, in the case of interruption of model training, the waste of computing resources can be effectively reduced and the continuity and efficiency of the training process can be ensured.
Smart Images
Figure CN119648514A_ABST
Abstract
Description
Background Art
[0002] With the development of the times, intelligent models have also been deeply developed. In today's society, the application of intelligent models is becoming more and more widespread. However, during the training of intelligent models, due to factors such as huge training resources, complex data processing, and long training time, training interruptions may occur. After the training is interrupted, the training task usually needs to be restarted, resulting in a waste of computing resources. Summary of the invention
[0003] The present disclosure provides a data processing method, apparatus, device and storage medium, which at least to a certain extent reduce the waste of computing resources during the intelligent model training process.
[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0005] According to one aspect of the present disclosure, there is provided a data processing method, comprising:
[0006] Perform model training in GPU nodes to obtain parameter gradients and intermediate data;
[0007] Determine whether the intermediate data meets the preset conditions;
[0008] When the intermediate data meets the preset conditions, the intermediate data and parameter gradients are stored in the CPU node;
[0009] In the case of interruption of model training, the current model training stage is determined, and the model training stage includes the forward propagation stage and the backward propagation stage;
[0010] Context recovery is completed based on the model training stage and prediction strategy.
[0011] In one embodiment of the present disclosure, determining whether the intermediate data meets a preset condition includes:
[0012] Mask the intermediate data to obtain a mask vector;
[0013] When the mask vector is greater than the first preset threshold and the intermediate data is within a preset range, it is determined that the intermediate data meets the preset condition.
[0014] In one embodiment of the present disclosure, context recovery is completed based on the model training stage and the prediction strategy, including:
[0015] When the model training stage is the forward propagation stage, forward calculation and backward calculation are performed on the intermediate data according to the CPU node to complete the context recovery.
[0016] In one embodiment of the present disclosure, context recovery is completed based on the model training stage and the prediction strategy, including:
[0017] When the model training stage is the back propagation stage, the intermediate data is filtered to obtain filtered intermediate data;
[0018] Forward and backward calculations are performed based on the CPU nodes, filtered intermediate data, and gradient parameters to complete context recovery.
[0019] In one embodiment of the present disclosure, screening the intermediate data includes:
[0020] Dropout layer that disables gradient computation at intermediate data inputs.
[0021] In one embodiment of the present disclosure, the method further includes:
[0022] Assign corresponding CPU nodes to GPU nodes based on computing power.
[0023] In one embodiment of the present disclosure, the CPU node includes a CPU master node and a CPU computing node;
[0024] Perform forward and backward calculations based on the CPU nodes, filtered intermediate data, and gradient parameters to complete context recovery, including:
[0025] The CPU master node determines the computing strategy based on the computing resources of the CPU computing nodes;
[0026] The computing strategy is sent to the CPU computing node, so that the CPU computing node performs forward computing and backward computing according to the filtered intermediate data and gradient parameters to complete context recovery.
[0027] According to another aspect of the present disclosure, there is provided a data processing device, comprising:
[0028] The training module is used to perform model training in the GPU node to obtain parameter gradients and intermediate data;
[0029] A first determination module, used to determine whether the intermediate data meets a preset condition;
[0030] A storage module, which is used to store the intermediate data and parameter gradients to the CPU node when the intermediate data meets the preset conditions;
[0031] The second determination module determines the current model training stage when the model training is interrupted. The model training stage includes a forward propagation stage and a backward propagation stage.
[0032] The recovery module is used to complete context recovery based on the model training stage and prediction strategy.
[0033] In one embodiment of the present disclosure, the first determining module includes:
[0034] A processing unit, used for performing mask processing on the intermediate data to obtain a mask vector;
[0035] The determination unit is used to determine whether the intermediate data meets a preset condition when the mask vector is greater than a first preset threshold and the intermediate data is within a preset range.
[0036] In one embodiment of the present disclosure, the recovery module includes:
[0037] A screening unit, used for screening the intermediate data to obtain screened intermediate data when the model training stage is a back propagation stage;
[0038] The computing unit is used to perform forward and backward calculations based on the CPU nodes, the filtered intermediate data, and the gradient parameters to complete context recovery.
[0039] In one embodiment of the present disclosure, the screening unit comprises:
[0040] Input subunit, used to input the Dropout layer that prohibits gradient calculation at the intermediate data input.
[0041] In one embodiment of the present disclosure, the device further includes:
[0042] The allocation module is used to allocate corresponding CPU nodes to GPU nodes based on the computing power.
[0043] In one embodiment of the present disclosure, the CPU node includes a CPU master node and a CPU computing node;
[0044] Computing unit, including:
[0045] Determine the subunits so that the CPU master node determines the computing strategy based on the computing resources of the CPU computing node;
[0046] The sending subunit is used to send the calculation strategy to the CPU calculation node, so that the CPU calculation node performs forward calculation and backward calculation according to the filtered intermediate data and gradient parameters to complete context recovery.
[0047] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-mentioned data processing method by executing the executable instructions.
[0048] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.
[0049] The data processing method, apparatus, device and storage medium provided by the embodiments of the present disclosure perform model training in a GPU node to obtain parameter gradients and intermediate data, determine whether the intermediate data meets preset conditions, and store the intermediate data and parameter gradients in a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0050] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.
[0052] Figure 1 A flow chart of a data processing method in an embodiment of the present disclosure is shown;
[0053] Figure 2 Another data processing method flow chart in the embodiment of the present disclosure is shown;
[0054] Figure 3 A flow chart of another data processing method in an embodiment of the present disclosure is shown;
[0055] Figure 4 A flow chart of another data processing method in an embodiment of the present disclosure is shown;
[0056] Figure 5 A data processing device architecture diagram according to an embodiment of the present disclosure is shown;
[0057] Figure 6 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0059] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0060] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0061] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0062] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0063] During the training process of large-scale deep learning models, due to factors such as huge resources, complex data processing, and long running time, there may be frequent interruptions. Large-scale deep learning models usually require a large amount of computing resources, including high-performance GPU or TPU clusters. The more training nodes there are, the longer it takes, and the greater the probability of node failure during training. Large-scale training jobs will inevitably encounter node failures.
[0064] In related technologies, checkpoint technology is usually used for fault tolerance and recovery during training. However, in the checkpoint technology, the cluster needs to be constantly paused to save the current training context. And when reloading from the checkpoint, usually only one generation of context for multiple training iterations can be saved, which may cause context loss.
[0065] In order to solve the above problems, the embodiments of the present disclosure provide a data processing method, apparatus, device and storage medium.
[0066] Figure 1 A flow chart of a data processing method in an embodiment of the present disclosure is shown.
[0067] like Figure 1 As shown, the method may include:
[0068] S110, performing model training in the GPU node to obtain parameter gradients and intermediate data.
[0069] In some embodiments, intermediate data may include tensors generated by operations at each level in the model.
[0070] In some embodiments, each GPU node is configured with a CPU connected to the GPU. The CPU includes a CPU master node and a CPU computing node.
[0071] The CPU master node can be the CPU with the shortest connection distance to the GPU.
[0072] In some embodiments, the CPU master node is connected to the GPU and the CPU computing node respectively.
[0073] S120, determining whether the intermediate data meets a preset condition.
[0074] In some embodiments, whether the intermediate data meets a preset condition may be determined based on the amount of computation required to obtain the intermediate data.
[0075] The method for determining the amount of computation of intermediate data is not specifically limited in the present disclosure.
[0076] S130, when the intermediate data meets the preset conditions, the intermediate data and the parameter gradient are stored in the CPU node.
[0077] In some embodiments, storing the intermediate data and parameter gradients to the CPU node may include:
[0078] The intermediate data and the parameter gradient are sent to the CPU master node so that the CPU master node sends the intermediate data and the parameter gradient to the CPU computing node respectively.
[0079] In some embodiments, by screening the intermediate data, the problem of low training efficiency caused by excessive amount of data stored in the CPU node is avoided.
[0080] In some embodiments, when the intermediate data does not meet a preset condition, only the parameter gradient may be stored in the CPU node.
[0081] S140, when the model training is interrupted, determining the current model training stage, where the model training stages include a forward propagation stage and a backward propagation stage.
[0082] In some embodiments, forward propagation and back propagation are two core stages of model training. Forward propagation is responsible for converting input data into prediction results, while back propagation is responsible for adjusting model parameters according to the error between the prediction results and the true value.
[0083] The present disclosure does not specifically limit the method for determining whether the model training stage is the forward propagation stage or the backward propagation stage.
[0084] In some embodiments, parameter gradients may include weights and biases of the model.
[0085] S150, completing context recovery based on the model training stage and prediction strategy.
[0086] In some embodiments, completing context recovery includes continuing the model training process after model training is interrupted.
[0087] The data processing method provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0088] Figure 2 A flow chart of another data processing method in an embodiment of the present disclosure is shown.
[0089] like Figure 2 As shown, the method may include:
[0090] S210, performing model training in the GPU node to obtain parameter gradients and intermediate data;
[0091] S220, performing mask processing on the intermediate data to obtain a mask vector.
[0092] In some embodiments, masking the intermediate data may include performing masking based on KV cache optimization technology (Heavy-Hitter Oracle, H2O).
[0093] In some embodiments, the intermediate data may be a generated tensor newtensor.
[0094] In some embodiments, the intermediate data may be masked to obtain H2O_mask[seq(newtensor)].
[0095] S230: When the mask vector is greater than a first preset threshold and the intermediate data is within a preset range, determine that the intermediate data meets a preset condition.
[0096] In some embodiments, the present disclosure does not specifically limit the size of the first preset threshold.
[0097] In some embodiments, the first preset threshold may be H2O_MIN.
[0098] The preset range may be determined based on the amount of calculation required to determine the intermediate data.
[0099] In some embodiments, different preset ranges may be determined based on different levels of the model-generated intermediate data.
[0100] For example, the output of softmax, the mask of dropout, and the output of dropout. These three have nothing to do with matrix multiplication and require very little computation, so they are put into the recompute set: re_compute_set.
[0101] The intermediate output of the linear layer, Q, K, V and other intermediate tensors, are placed in the communication set comm_set because there is a matrix multiplication with W in front of it, which requires a lot of calculation.
[0102] In some embodiments, during the model training process, the method further includes sending H2O_mask from the GPU to the CPU.
[0103] S240, when the intermediate data meets the preset conditions, storing the intermediate data and the parameter gradient to the CPU node;
[0104] S250, in the case where the model training is interrupted, determining the current model training stage, where the model training stage includes a forward propagation stage and a backward propagation stage;
[0105] S260, completing context recovery based on the model training stage and the prediction strategy.
[0106] The data processing method provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0107] Figure 3 A flow chart of another data processing method in an embodiment of the present disclosure is shown.
[0108] like Figure 3 As shown, the method may include:
[0109] S310, performing model training in the GPU node to obtain parameter gradients and intermediate data;
[0110] S320, determining whether the intermediate data meets a preset condition;
[0111] S330, when the intermediate data meets the preset conditions, storing the intermediate data and the parameter gradient to the CPU node;
[0112] S340, in the case where the model training is interrupted, determining the current model training stage, where the model training stage includes a forward propagation stage and a backward propagation stage;
[0113] S350, when the model training stage is the forward propagation stage, forward calculation and backward calculation are performed on the intermediate data according to the CPU node to complete context recovery.
[0114] In some embodiments, performing forward calculation and backward calculation on the intermediate data according to the CPU node may include loading the intermediate data and gradient parameters according to batch division of the secondary data parallelism, and then performing forward calculation and backward calculation.
[0115] Among them, batch processing is a common data processing technology in computer science, especially widely used in the fields of big data processing and machine learning. Through batch processing, a large amount of data can be divided into multiple smaller batches, and then these batches are processed in parallel or serially, thereby improving data processing efficiency.
[0116] In some embodiments, the CPU central node may summarize the calculation results after the secondary data parallelization.
[0117] The data processing method provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0118] Figure 4 A flow chart of another data processing method in an embodiment of the present disclosure is shown.
[0119] like Figure 4 As shown, the method may include:
[0120] S410, performing model training in the GPU node to obtain parameter gradients and intermediate data;
[0121] S420, determining whether the intermediate data meets a preset condition;
[0122] S430, when the intermediate data meets the preset conditions, storing the intermediate data and the parameter gradient to the CPU node;
[0123] S440, in the case where the model training is interrupted, determining the current model training stage, where the model training stage includes a forward propagation stage and a backward propagation stage;
[0124] S450, when the model training stage is the back propagation stage, the intermediate data is screened to obtain screened intermediate data.
[0125] In some embodiments, screening the intermediate data includes:
[0126] Dropout layer that disables gradient computation at intermediate data inputs.
[0127] In some embodiments, filtering the intermediate data may further include filtering based on the amount of calculation of the intermediate data.
[0128] Exemplarily, a Dropout layer that prohibits gradient calculation at intermediate data input may include:
[0129] A pseudo dropout layer labeled no grad is added after each intermediate data, where no grad is used to disable gradient calculation.
[0130] S460, forward calculation and backward calculation are performed according to the CPU nodes, the filtered intermediate data, and the gradient parameters to complete the context recovery.
[0131] In some embodiments, performing forward calculation and backward calculation according to the CPU node, the filtered intermediate data, and the gradient parameter, completing context recovery may include:
[0132] Enable the CPU master node to determine the computing strategy based on the computing resources of the CPU computing node;
[0133] The computing strategy is sent to the CPU computing node, so that the CPU computing node performs forward computing and backward computing according to the filtered intermediate data and gradient parameters to complete context recovery.
[0134] In some embodiments, the method may further include:
[0135] Assign corresponding CPU nodes to GPU nodes based on computing power.
[0136] Exemplarily, N vCPUs are allocated to each GPU, where N is the computing power ratio of the GPU to the vCPU.
[0137] The data processing method provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0138] Based on the same inventive concept, the present disclosure also provides a data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.
[0139] Figure 5 A structural diagram of a data processing device in an embodiment of the present disclosure is shown.
[0140] like Figure 5 As shown, the apparatus 500 may include:
[0141] A training module 510 is used to perform model training in a GPU node to obtain parameter gradients and intermediate data;
[0142] A first determination module 520, used to determine whether the intermediate data meets a preset condition;
[0143] The storage module 530 is used to store the intermediate data and the parameter gradient to the CPU node when the intermediate data meets the preset conditions;
[0144] A second determination module 540, in the case where the model training is interrupted, determines the current model training stage, where the model training stage includes a forward propagation stage and a backward propagation stage;
[0145] The recovery module 550 is used to complete context recovery based on the model training stage and the prediction strategy.
[0146] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0147] In one embodiment of the present disclosure, the first determining module includes:
[0148] A processing unit, used for performing mask processing on the intermediate data to obtain a mask vector;
[0149] The determination unit is used to determine whether the intermediate data meets a preset condition when the mask vector is greater than a first preset threshold and the intermediate data is within a preset range.
[0150] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0151] In one embodiment of the present disclosure, the recovery module includes:
[0152] A screening unit, used for screening the intermediate data to obtain screened intermediate data when the model training stage is a back propagation stage;
[0153] The computing unit is used to perform forward and backward calculations based on the CPU nodes, the filtered intermediate data, and the gradient parameters to complete context recovery.
[0154] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0155] In one embodiment of the present disclosure, the screening unit comprises:
[0156] Input subunit, used to input the Dropout layer that prohibits gradient calculation at the intermediate data input.
[0157] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0158] In one embodiment of the present disclosure, the device further includes:
[0159] The allocation module is used to allocate corresponding CPU nodes to GPU nodes based on the computing power.
[0160] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0161] In one embodiment of the present disclosure, the CPU node includes a CPU master node and a CPU computing node;
[0162] Computing unit, including:
[0163] Determine the subunits so that the CPU master node determines the computing strategy based on the computing resources of the CPU computing node;
[0164] The sending subunit is used to send the calculation strategy to the CPU calculation node, so that the CPU calculation node performs forward calculation and backward calculation according to the filtered intermediate data and gradient parameters to complete context recovery.
[0165] The data processing device provided by the embodiments of the present disclosure performs model training in a GPU node to obtain parameter gradients and intermediate data, determines whether the intermediate data meets preset conditions, and stores the intermediate data and parameter gradients to a CPU node when the intermediate data meets the preset conditions. When model training is interrupted, the current stage of model training is determined, and the stages of model training include a forward propagation stage and a backward propagation stage. Context recovery is completed based on the stages of model training and preset strategies. When model training is interrupted, the waste of computing resources in model training is reduced.
[0166] The data processing device provided in the embodiments of the present disclosure can be used to execute the positioning methods provided in the above-mentioned method embodiments. The implementation principles and technical effects thereof are similar and will not be described in detail here for the sake of brevity.
[0167] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".
[0168] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0169] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).
[0170] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps of the above method embodiment:
[0171] Perform model training in GPU nodes to obtain parameter gradients and intermediate data;
[0172] Determine whether the intermediate data meets the preset conditions;
[0173] When the intermediate data meets the preset conditions, the intermediate data and parameter gradients are stored in the CPU node;
[0174] In the case of interruption of model training, the current model training stage is determined, and the model training stage includes the forward propagation stage and the backward propagation stage;
[0175] Context recovery is completed based on the model training stage and prediction strategy.
[0176] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .
[0177] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0178] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0179] The electronic device 600 may also communicate with one or more external devices 640 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 6100. As shown, the network adapter 6100 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0180] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.
[0181] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.
[0182] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0183] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0184] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.
[0185] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).
[0186] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0187] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.
[0188] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.
[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A data processing method, characterized in that: include: Perform model training in GPU nodes to obtain parameter gradients and intermediate data; Determining whether the intermediate data meets a preset condition; When the intermediate data meets the preset conditions, the intermediate data and the parameter gradient are stored in the CPU node; In the case where the model training is interrupted, determining the current model training stage, wherein the model training stage includes a forward propagation stage and a backward propagation stage; Context recovery is completed based on the model training stage and prediction strategy.
2. The method according to claim 1, characterized in that The determining whether the intermediate data meets a preset condition includes: Performing mask processing on the intermediate data to obtain a mask vector; When the mask vector is greater than a first preset threshold and the intermediate data is within a preset range, it is determined that the intermediate data meets a preset condition.
3. The method according to claim 1, characterized in that The model-based training phase and prediction strategy complete context recovery, including: When the model training stage is the forward propagation stage, forward calculation and backward calculation are performed on the intermediate data according to the CPU node to complete context recovery.
4. The method according to claim 1, characterized in that: The model-based training phase and prediction strategy complete context recovery, including: When the model training stage is a back propagation stage, the intermediate data is screened to obtain screened intermediate data; Forward and backward calculations are performed based on the CPU nodes, filtered intermediate data, and gradient parameters to complete context recovery.
5. The method according to claim 4, characterized in that The screening of the intermediate data includes: Dropout layer that disables gradient computation at the intermediate data input.
6. The method according to claim 1, characterized in that The method further comprises: Assign corresponding CPU nodes to GPU nodes based on computing power.
7. The method according to claim 4 or 6, characterized in that: The CPU nodes include a CPU master node and a CPU computing node; The forward calculation and the backward calculation are performed according to the CPU nodes, the filtered intermediate data and the gradient parameters to complete the context recovery, including: Enable the CPU master node to determine a computing strategy based on the computing resources of the CPU computing node; The computing strategy is sent to the CPU computing node, so that the CPU computing node performs forward computing and backward computing according to the filtered intermediate data and the gradient parameters to complete context recovery.
8. A data processing device, characterized in that: include: The training module is used to perform model training in the GPU node to obtain parameter gradients and intermediate data; A first determination module, used to determine whether the intermediate data meets a preset condition; A storage module, for storing the intermediate data and parameter gradients in a CPU node when the intermediate data meets a preset condition; A second determination module, in the case where the model training is interrupted, determines the stage of the current model training, wherein the stage of the model training includes a forward propagation stage and a backward propagation stage; The recovery module is used to complete context recovery based on the model training stage and prediction strategy.
9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.