Model deployment method, data processing method, and electronic device

Abstract

The present disclosure provides a model deployment method, a data processing method and an electronic device. The model deployment method comprises: determining N data processing stages of a target model, the target model being divided into N data processing stages and running in a pipeline parallel manner; determining, according to a stage number of the data processing stage, whether the data processing stage is a first type data processing stage or a second type data processing stage in a warm-up stage, the first type data processing stage being an active data transfer party, and the second type data processing stage being an active data receiving party; and deploying the first type data processing stage and the second type data processing stage in one computing node, the computing node comprising a plurality of computing devices, the plurality of computing devices comprising a first type computing device where the first type data processing stage is located and a second type computing device where the second type data processing stage is located. The embodiments of the present disclosure can reduce the deployment cost of a large model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of large model, in particular, to a model deployment method, a data processing method and an electronic device. BACKGROUND

[0002] A large model (LLM) running in a pipeline parallel manner needs to save all activation data in a forward propagation process in an early warm-up stage for use in a backward propagation stage, which requires a large memory and puts forward a high requirement on the memory capacity of a computing device such as a GPU running the large model, resulting in a high cost of large model deployment.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The embodiments of the present disclosure are used to reduce the requirement on the memory capacity of a hardware device in a model training stage and reduce the cost of large model deployment.

[0005] According to a first aspect of an embodiment of the present disclosure, a model deployment method is provided, comprising: determining N data processing stages of a target model, the target model being divided into N data processing stages and running in a pipeline parallel manner; determining whether the data processing stage is a first type data processing stage or a second type data processing stage according to the stage number of the data processing stage in a warm-up stage, the first type data processing stage being an activation data transfer side, and the second type data processing stage being an activation data receiving side; deploying the first type data processing stage and the second type data processing stage in one computing node, the computing node comprising a plurality of computing devices, the plurality of computing devices comprising a first type computing device where the first type data processing stage is located and a second type computing device where the second type data processing stage is located.

[0006] According to a second aspect of an embodiment of the present disclosure, a data processing method is provided, executed by a target model deployed by the model deployment method according to any one of the above, the data processing method comprising: receiving and processing training data in response to a training data input message; when it is determined that a warm-up stage is entered, transmitting activation data of a first type data processing stage from a first type computing device to a second type computing device for temporary storage in one computing node, the first type computing device being a computing device where the first type data processing stage is deployed, and the second type computing device being a computing device where a second type data processing stage is deployed; and in a backward propagation stage, taking the activation data from the second type computing device back to the first type computing device.

[0007] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising a memory and a processor coupled to the memory, the processor is configured to execute the method according to any one of the preceding method embodiments based on instructions stored in the memory.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a program, the program is executed by a processor to implement the model deployment method according to any one of the preceding method embodiments.

[0009] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises a computer program, the computer program is executed by a processor to implement the steps of the method according to any one of the preceding method embodiments.

[0010] The embodiments of the present disclosure can allow the large model to transfer the activation data from the first type of computing device where the first type of data processing stage is located to the second type of computing device where the second type of data processing stage is located in the preheating stage of the training process by determining the data processing stage with higher memory requirement in the preheating stage as the first type of data processing stage and determining the data processing stage with relatively lower memory requirement in the preheating stage as the second type of data processing stage, and deploying the first type of data processing stage and the second type of data processing stage in the same computing node, to realize the memory sharing between different computing devices in the same computing node, thereby realizing the low communication cost memory transfer, reducing the memory requirement of the large model deployment to the computing device, and reducing the large model deployment cost.

[0011] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor based on these drawings.

[0013] Figure 1 is a schematic diagram of a data processing system running a target model.

[0014] Figure 2 is a schematic diagram of the running principle of a target model.

[0015] Figure 3 is a flowchart of the model deployment method in the exemplary embodiments of the present disclosure.

[0016] Figure 4is a sub-process of step S13 in one embodiment of the present disclosure.

[0017] Figure 5 is Figure 4 is a schematic diagram corresponding to the embodiment shown.

[0018] Figure 6 is a schematic diagram of a computing device group in an exemplary embodiment of the present disclosure.

[0019] Figure 7 is a flowchart of a data processing method in an embodiment of the present disclosure.

[0020] Figure 8 is a sub-flowchart of step S72 in one embodiment of the present disclosure.

[0021] Figure 9 is a schematic diagram of the model in which the method 700 is run. Figure 2

[0022] Figure 10 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Example implementations are now described with reference to the following drawings. The example implementations, however, can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the example implementations to those skilled in the art. The features, structures, or characteristics described can be combined in one or more implementations. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, etc., to provide a thorough understanding of the example implementations. One skilled in the relevant art will recognize, however, that the example implementations can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. Some features, structures, or characteristics can be described as being implemented over one or more entities, portions of which can be implemented as hardware, software, or firmware. In other examples, some features, structures, or characteristics can also be implemented to locate across multiple devices or components.

[0024] In addition, the accompanying drawings are included to provide a thorough understanding of the present disclosure and are incorporated in and constitute a part of this specification. The same references refer to the same or similar

[0025] Example implementations of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0026] ​To facilitate the explanation of the principle, first introduce the hardware system of deploying large models.

[0027] Figure 1 is a schematic diagram of a data processing system running a target model.

[0028] Reference Figure 1 In the embodiments of the present disclosure, the data processing system 100 can include:

[0029] At least one computing node 1, each computing node 1 including a plurality of computing devices 11 and at least one central processor 12 interacting with the plurality of computing devices 11, the plurality of computing devices 11 exchanging data through a high-speed data transmission channel.

[0030] Each computing device 11 is provided with a data processing stage Si of a target model, the target model being a large model (LLM), the target model being run through a pipeline parallel manner and being divided into N data processing stages, 1≤i≤N.

[0031] The computing node 1 includes but is not limited to a server, a single workstation in a workstation cluster. The computing device 11 is, for example, a GPU (Graphics Processing Unit) or an ASIC (Application-Specific Integrated Circuit) such as a TPU (Tensor Processing Unit) provided in the computing node 1. Alternatively, the computing device 11 can also be implemented by an FPGA (Field Programmable Gate Array).

[0032] During the running of the target model, data is transferred in sequence in the stages responsible by each computing device. After the previous computing device completes its task of the corresponding data processing stage Si, it quickly transfers the processing result to the next computing device for further processing. Within the same computing node 1, the computing devices 11 communicate with each other through a high-speed data transmission channel, such as an NVLink link. Between different computing nodes 1, the computing devices 11 communicate with each other through a high-speed network interconnection architecture, such as an InfiniBand network or a high-speed Ethernet technology. Among them, NVLink is a high-speed, high-bandwidth interconnection technology developed by NVIDIA. It is mainly used to connect multiple GPUs to realize direct communication between GPUs, and its purpose is to accelerate data transmission between multiple GPUs and overcome the problem of insufficient bandwidth of traditional PCI-Express (PCI-e) bus in multi-GPU communication scenarios.

[0033] Therefore, the data transmission between the computing devices 11 within the same computing node 1 has a significantly higher transmission speed than the data transmission between the computing devices 11 of different computing nodes 1.

[0034] In an exemplary embodiment, the data processing system 100 can include a memory management module. The memory management module is responsible for reasonably allocating and scheduling the memory resources of each computing node and each computing device to meet the data storage and reading needs of the large model in different data processing stages. The memory management module can be set on the computing device 11 of a certain computing node 1, or it can exist in the data processing system 100 as a special control unit independent of each computing device 11 to control the data transmission between multiple computing nodes 1.

[0035] The data processing method provided by the embodiments of the present disclosure can be implemented by the memory management module or by the target model by setting the running mode of the target model.

[0036] Figure 2 is a schematic diagram of the running principle of the target model.

[0037] Referring to Figure 2 , the target model is divided into N (N = 4 in the Figure 2 embodiments) data processing stages (stages), and runs in a flow parallel manner and a 1F1B (One-Forward-One-Backward, one forward propagation-one backward propagation strategy) mode.

[0038] In the forward propagation stage, data flows through each subsequent stage in turn from the first data processing stage S1. The data processing stage S1 generally performs preliminary feature extraction and conversion on the input data, such as word vector encoding for text data or basic feature extraction for image data, and then passes the processed intermediate data to the data processing stage S2. The data processing stage S2 further processes based on the results of the data processing stage S1, which may involve more complex feature fusion or semantic understanding related calculations, and so on, until the data completes the last forward processing step in the data processing stage SN, obtaining the forward output result of the model.

[0039] Referring to Figure 2 , it is shown that Figure 2 one square represents the processing of one micro-batch data by one data processing stage, F represents the forward propagation stage processing, B represents the backward propagation stage, and the first value in the subscript represents the data processing stage and the second value represents the batch number, for example, F 2,1This represents the second data processing stage S2, which involves the forward propagation of the second batch of data. It can also be described as the forward propagation of the second batch of data reaching the second data processing stage S2. Correspondingly, B... 2,1 This represents the second data processing stage S2, where the second batch of data undergoes backpropagation processing. It can also be described as the backpropagation processing of the second batch of data reaching the second data processing stage S2. The process of the first batch of data being completely processed is F. 1,1 →F 2,1 →F 3,1 →F 4,1 →B 4,1 →B 3,1 →B 2,1 →B 1,1 ,like Figure 2 As indicated by the middle arrow. The processing procedure for other batches of data is the same and will not be described again.

[0040] During the forward propagation phase of the model, the intermediate results calculated at each data processing stage are called activation data. This activation data contains feature information that the model gradually extracts during the processing of input data. For example, in a simple multilayer neural network, after the input layer receives data, it multiplies it with the weights of the neurons in the first layer, adds a bias, and then passes it through an activation function. The resulting output is the activation data for the first layer. This data will then be used as input for the next layer. During model training, each data processing stage in the forward propagation phase needs to save the activation data for subsequent model training.

[0041] After completing one forward propagation, backpropagation immediately follows, starting from the data processing stage SN. Data processing stage SN calculates its own gradient with respect to the loss function and passes it to data processing stage SN-1. Data processing stage SN-1 uses the received gradient information, combined with its own computational logic (model parameters) from the forward propagation and the saved activation data, to calculate its own gradient and continue to pass it back to data processing stage SN-2. In this process, each stage updates its own model parameters to optimize the model, enabling it to gradually learn the patterns and features in the data.

[0042] exist Figure 2 In the example, the data processing stage S2 is in F 2,1 When generating F 2,1 The activation data can then be used in subsequent B 2,1 Use F when 2,1 The activation data updates the parameters of data processing stage S2. For data processing stage S2, after completing B... 2,1 Previously, it was necessary to temporarily store F. 2,1 Activation data.

[0043] Therefore, during the running of the 1F1B mode, there are a warm-up phase, a steady-state stage and a cool-down phase. In the warm-up phase, each data processing stage only performs forward propagation processing without backward propagation processing, and the generated activation data is continuously increasing, and the memory pressure in each computing device 11 is increasing; in the steady-state stage, the data processing stages with later serial numbers start to perform backward propagation processing using activation data, and then delete the activation data, while other data processing stages are still generating new activation data, and the memory pressure of each computing device remains basically unchanged; in the cool-down phase, each data processing stage has completed the forward propagation processing and starts to perform only backward propagation processing, which continuously consumes the temporarily stored activation data, so that the memory pressure of each computing device gradually decreases. After the end of the cool-down phase, the cumulative gradient of each micro-batch is used to update the model parameters of each data processing stage.

[0044] Observation Figure 2 It can be seen that the data processing stage S1 must store the activation data of the same number of micro-batches as the pipeline parallelism in the warm-up phase. In other data processing stages, the number of micro-batches processed in the warm-up phase is linearly reduced, and the last data processing stage S4 only maintains the activation data of one micro-batch. Therefore, the imbalance of memory usage exists in the entire pipeline stage (S1-S4), and the imbalance degree is amplified with the upgrade of pipeline parallelism, i.e., the increase of the number of data processing stages.

[0045] wherein the memory occupied by the data processing stage Si on the computing device is M(i)=W(i)+A(i), W(i) represents the size of the model parameters including the optimizer state, which is a constant value W0; A(i) represents the memory occupation value of the saved activation data, MB_num(i) is the number of micro-batches that need to save the activation parameters, and A0 is the memory occupation value of the activation data corresponding to each micro-batch data. When each data processing stage has the same number of layers, A0 is a constant, and M(i) depends on the number of micro-batches MB_num(i).

[0046] Therefore, due to the large memory pressure of the warm-up phase, the large model requires a higher memory of the computing device, which causes pressure on the deployment cost of the large model.

[0047] To this end, the model deployment method 300 in the embodiments of the present disclosure solves the problem.

[0048] Figure 3 is a flowchart of the model deployment method in the exemplary embodiments of the present disclosure.

[0049] Reference Figure 3 The model deployment method 300 can include the following steps.

[0050] In step S11, N data processing stages of a target model are determined, and the target model is run in a flow water parallel mode and is divided into N data processing stages.

[0051] In step S12, it is determined according to the stage number of the data processing stage whether the data processing stage is a first type data processing stage or a second type data processing stage in the warm-up stage, the first type data processing stage is an active data transfer party, and the second type data processing stage is an active data receiving party.

[0052] In step S13, the first type data processing stage and the second type data processing stage are deployed in one computing node, and the computing node includes a plurality of computing devices, the plurality of computing devices include a first type computing device where the first type data processing stage is located and a second type computing device where the second type data processing stage is located.

[0053] The embodiments of the present disclosure determine the data processing stage with high memory requirement in the warm-up stage as the first type data processing stage, and determine the data processing stage with relatively low memory requirement in the warm-up stage as the second type data processing stage, and then deploy the first type data processing stage and the second type data processing stage in the same computing node. The active data can be transferred from the first type computing device where the first type data processing stage is located to the second type computing device where the second type data processing stage is located in the warm-up stage of the training process of the large model in the same computing node, the memory sharing between different computing devices in the same computing node is realized, the memory transfer with low communication cost is realized, the memory requirement of the large model deployment to the computing device is reduced, and the large model deployment cost is reduced.

[0054] Next, the method 300 will be described in detail.

[0055] In step S11, N data processing stages of a target model are determined.

[0056] Determining the N data processing stages is mainly used to determine the number N of data processing stages and the processing order corresponding to each data processing stage, and then set the sequence number for the data processing stage, and set the data processing stage with processing order i as data processing stage Si, 1≤i≤N.

[0057] In step S12, it is determined according to the stage number of the data processing stage whether the data processing stage is a first type data processing stage or a second type data processing stage in the warm-up stage, the first type data processing stage is an active data transfer party, and the second type data processing stage is an active data receiving party.

[0058] As shown in Figure 2 , when the number of micro-batches m handled by the data processing stage Si is greater than the number of data processing stages N, for example Figure 2 F 1,5 in

[0059] (1)

[0060] (2)

[0061] The memory decreases linearly with the increase of the data processing stage sequence number, and more memory is occupied when i≤N / 2, and less memory is occupied in subsequent stages. That is, the first half of the data processing stage, because more activation data of the micro-batch needs to be saved to wait for backward propagation, the requirement for memory is higher; for the second half of the data processing stage, because the backward propagation has started, a part of the activation data is consumed, and the requirement for memory starts to decrease.

[0062] Therefore, in the example embodiment, step S12 can include:

[0063] When i<(N+1) / 2, the i-th data processing stage is the first type of data processing stage;

[0064] When i≥(N+1) / 2, the i-th data processing stage is the second type of data processing stage;

[0065] Wherein, N≥1, 1≤i≤N.

[0066] Wherein, the first type of data processing stage is the activation data transfer party, which is set to transfer the generated activation data out, so as to reduce the memory pressure on the computing device 11 where it is located; the second type of data processing stage is the activation data receiving party, which is set to receive the activation data from the first type of data processing stage and temporarily store it, and allow the first type of data processing stage to retrieve the activation data when it needs to use it in the backward propagation stage, so as to share the memory pressure from other computing devices 11 by using the idle memory capacity, solve the problem of high memory pressure and high requirement on computing devices 11 caused by the preheating stage, reduce the memory requirement on each computing device 11, and reduce the deployment cost of large models.

[0067] The above example can be briefly described as balancing the memory occupied by the activation data of MB_num(i) micro-batches and the activation data of MB_num(N-i-1) micro-batches by transferring the activation data of the data processing stages before and after N / 2 to the same node for activation transmission.

[0068] So that:

[0069] (3)

[0070] The sum of the saved maximum micro-batch numbers in the matching stage is:

[0071] (4)

[0072] The number of micro-batches in the optimal balance in the stable stage is N+2.

[0073] (5)

[0074] The optimal memory balance is:

[0075] (6)

[0076] Select all data processing stages in which the saved micro-batch number MB_num(i) is less than or equal to the optimal micro-batch number MB_num(opt) as the first type of data processing stage, and send at most (N-i)-MB_num(opt)+1 micro-batches of activation data to the second type of data processing stage in the computing device. In this way, the first type of computing device and the second type of computing device that have an activation data transfer relationship can save at most i+1 micro-batches of activation data. This can be represented as: when MB_num(p-i-1)≤MB_num(opt), there is:

[0077] (7)

[0078] The above embodiment performs an exemplary division, which is divided according to the first half data processing stage / second half data processing stage. In other embodiments, the memory occupancy values of each data processing stage in the warm-up stage can be obtained according to the actual calculation, and it can be determined which data processing stages are the first type of data processing stage, which data processing stages are the second type of data processing stage, and which data processing stages are neither the first type of data processing stage nor the second type of data processing stage.

[0079] For example, assuming N = 8, the memory occupancy values of the first to eighth data processing stages in the warm-up stage are 100, 95, 90, 85, 80, 75, 70, and 65 storage units (data is only an example, and the storage unit is, for example, GB), each data processing stage corresponds to a computing device, and the available memory of each computing device is 90 (also for example, GB). At this time, the first and second data processing stages can be set as the first type of data processing stage, the seventh and eighth data processing stages can be set as the second type of data processing stage, 10 memory occupancy values corresponding to the active data are transferred from the first data processing stage to the computing device where the eighth data processing stage is located in the warm-up stage, and 5 memory occupancy values corresponding to the active data are transferred from the second data processing stage to the computing device where the seventh data processing stage is located. At the same time, the third, fourth, fifth, and sixth data processing stages are not specially set, so as to reduce the amount of active data transfer and reduce the system running pressure.

[0080] In the above example, the first and second data processing stages can also be set as the first type of data processing stage, and the eighth data processing stage can be set as the second type of data processing stage. In the warm-up stage, 10 memory occupancy values corresponding to the active data are transferred from the first data processing stage to the computing device where the eighth data processing stage is located, and 5 memory occupancy values corresponding to the active data are transferred from the second data processing stage to the computing device where the eighth data processing stage is located. In this way, the number of computing devices involved in the transfer of active data is reduced, and the system running pressure is reduced.

[0081] That is, in the embodiments of the present disclosure, each data processing stage can be set as the first type of data processing stage, the second type of data processing stage, or not specially set. The number of first type of data processing stages and the number of second type of data processing stages can be the same or different. That is, the number of first type of data processing stages can be greater than the number of second type of data processing stages, or the number of first type of data processing stages can be less than the number of second type of data processing stages, for example, the active data of one first type of data processing stage is transferred to two or more second type of data processing stages for temporary storage, so as to further reduce the requirement for the memory capacity of each computing device.

[0082] In other embodiments, the memory occupancy values of each data processing stage in the warm-up stage can also be used to set the first type of data processing stage and the second type of data processing stage to determine the requirement for the available memory of the computing device, so as to further reduce the hardware cost of the computing device.

[0083] With the above examples, the first to third data processing stages can be set as the first type of data processing stage, the sixth to eighth data processing stages can be set as the second type of data processing stage, and in the warm-up stage, the first data processing stage can transfer 20 memory occupation values of activation data to the computing device where the eighth data processing stage is located, the second data processing stage can transfer 15 memory occupation values of activation data to the computing device where the seventh data processing stage is located, and the third data processing stage can transfer 5 memory occupation values of activation data to the computing device where the sixth data processing stage is located, so that after the transfer, the maximum memory occupation value of each pair of computing devices where the data processing stages are located is 85, and then the available memory of each computing device is set to 85, the computing devices are deployed according to the available memory 85, and the deployment cost of the large model is reduced.

[0084] There can be many methods for setting the first type of data processing stage and the second type of data processing stage, for example, the number of transferred activation data can also be determined in units of indivisible minimum units of activation data, and then the first type of data processing stage and the second type of data processing stage can be determined, and the present disclosure does not make special limitations on this.

[0085] By transferring the activation data generated by the first type of data processing stage in the warm-up stage to the computing device where the second type of data processing stage is located, the memory requirement of the target model for the computing device in the warm-up stage can be reduced, and the deployment cost of the large model can be reduced.

[0086] Since data transfer needs to be performed between computing devices, in order to overcome the impact of the additional data transfer operation on efficiency, the present disclosure embodiment adjusts the correspondence between the computing device and the data processing stage to avoid the low data transfer efficiency and the reduction of the overall system operation efficiency caused by the setting of the computing device according to the sequence number of the data processing stage in the related art.

[0087] In step S13, the first type of data processing stage and the second type of data processing stage are deployed in one computing node, and the computing node includes a plurality of computing devices, and the plurality of computing devices include a first type of computing device where the first type of data processing stage is located and a second type of computing device where the second type of data processing stage is located.

[0088] In the embodiment of the present disclosure, each data processing stage is deployed on each computing device of a computing node according to the data transfer relationship rather than the sequence number of the data processing stage.

[0089] In an exemplary embodiment, half of the computing devices can be directly set to deploy the first type of data processing stage, and the other half of the computing devices can be set to deploy the second type of data processing stage. In other embodiments, more detailed settings can be made to better achieve the purpose of reducing the deployment cost of the large model.

[0090] Figure 4 is a sub-process of step S13 in one embodiment of the present disclosure.

[0091] Referring to Figure 4 , step S13 can include:

[0092] Step S131, determining the memory occupation value of each data processing stage in the warm-up stage according to the stage number of the data processing stage;

[0093] Step S132, obtaining the available memory of each computing device;

[0094] Step S133, determining the to-be-transferred memory value of the first type of computing device corresponding to the first type of data processing stage according to the difference between the memory occupation value of the first type of data processing stage and the available memory of the computing device;

[0095] Step S134, determining the receivable memory value of the second type of computing device corresponding to the second type of data processing stage according to the difference between the available memory of the computing device and the memory occupation value of the second type of data processing stage;

[0096] Step S135, setting the plurality of computing devices in one computing node as the first type of computing device and the second type of computing device respectively, deploying the first type of data processing stage on the first type of computing device, and deploying the second type of data processing stage on the second type of computing device, wherein the sum of the to-be-transferred memory values of the first type of computing device in the computing node is less than the sum of the receivable memory values of the second type of computing device.

[0097] In Figure 4 the embodiment, the memory occupation value of each data processing stage in the warm-up stage (as given in the example above) and the available memory of each computing device can be determined first, then it is determined which several data processing stages need to transfer out the activated data and how much activated data needs to be transferred out under the current hardware condition (the available memory limit of the computing device). In addition, it is also necessary to find the temporary storage computing device for the activated data transferred out, so it is necessary to determine which computing device of the data processing stage can accept the activated data transferred out, and to calculate how much activated data these computing devices can accept at most.

[0098] In the example embodiment, first, each data processing stage can be set as a first type of data processing stage (the memory occupancy value is greater than the available memory) and a second type of data processing stage (the memory occupancy value is less than the available memory) according to whether the memory occupancy value in the pre-warming stage is greater than the available memory of the computing device. Next, according to the gap between the memory occupancy value and the available memory, it is determined how much active data (according to the memory occupancy value, referred to as the transfer memory value) needs to be transferred out of the first type of data processing stage in the pre-warming stage, and how much active data (according to the memory occupancy value, referred to as the receivable memory value) can be accepted by the second type of data processing stage in the pre-warming stage. Finally, the first type of data processing stage and the second type of data processing stage are deployed on the computing devices of the same computing node, and it is necessary to ensure that the sum of the transfer memory values corresponding to the first type of computing device in each computing node is less than the sum of the receivable memory values corresponding to the second type of computing device.

[0099] Figure 5 is Figure 4 An example embodiment corresponds to a schematic diagram.

[0100] Referring to Figure 5 , there are a computing node 51 and a computing node 52, each of which is provided with 8 computing devices, which are computing devices 511-518 and computing devices 521-528, respectively. The available memory value of each computing device is 10 GB. The target model has a total of 16 data processing stages, which are data processing stages S1-S16. The memory occupancy value of each data processing stage in the pre-warming stage decreases in order of the serial number, which is 15 GB, 14 GB, 13 GB, 12 GB, 11 GB, 10 GB, 9 GB, 8 GB, 7 GB, 6 GB, 5 GB, 4 GB, 3 GB, 2 GB, 1 GB, and 0.5 GB (the data is only an example and has no practical significance).

[0101] Therefore, the memory occupancy values of data processing stages S1-S5 are greater than 10 GB, which belong to the first type of data processing stage. The transfer memory value of data processing stage S1 is 15-10=5 GB, the transfer memory value of data processing stage S2 is 14-10=4 GB, the transfer memory value of data processing stage S3 is 13-10=3 GB, the transfer memory value of data processing stage S4 is 12-10=2 GB, and the transfer memory value of data processing stage S5 is 11-10=1 GB.

[0102] The memory occupancy values of data processing stages S7-S16 are less than 10 GB, which belong to the second type of data processing stage. Among them:

[0103] The receivable memory value of data processing stage S7 is 10-9=1 GB;

[0104] The acceptable memory value of the data processing stage S8 is 10-8=2GB;

[0105] The acceptable memory value of the data processing stage S9 is 10-7=3GB;

[0106] The acceptable memory value of the data processing stage S10 is 10-6=4GB;

[0107] The acceptable memory value of the data processing stage S11 is 10-5=5GB;

[0108] The acceptable memory value of the data processing stage S12 is 10-4=6GB;

[0109] The acceptable memory value of the data processing stage S13 is 10-3=7GB;

[0110] The acceptable memory value of the data processing stage S14 is 10-2=8GB;

[0111] The acceptable memory value of the data processing stage S15 is 10-1=9GB;

[0112] The acceptable memory value of the data processing stage S16 is 10-0.5=9.5GB.

[0113] In the computing node 51, if the computing device 511 is deployed S1, the to-be-transferred memory value thereof is 5GB; the computing device 512 is deployed S7, and the acceptable memory value thereof is 1GB. To ensure that the sum of the to-be-transferred memory values of the first type of computing devices in the computing node 51 is less than the sum of the acceptable memory values of the second type of computing devices, the deployment of other data processing stages on the remaining computing devices needs to be reasonably arranged. For example, if the computing device 513 is deployed S2, the to-be-transferred memory value thereof is 4GB, then a sufficient number of second type of data processing stages with larger acceptable memory values need to be deployed on subsequent computing devices, such as the computing device 514 is deployed S10, the acceptable memory value thereof is 4GB, and so on. Through continuous adjustment and combination, the memory allocation of the entire computing node 51 is balanced.

[0114] Similarly, in the computing node 52, the deployment planning is performed according to the memory occupation characteristics of each data processing stage. For example, if the computing device 521 is deployed S3, the to-be-transferred memory value thereof is 3GB, then the computing device 522 is deployed S8, and the acceptable memory value thereof is 2GB. The deployment on the subsequent computing devices needs to continue to be comprehensively considered to meet the memory allocation requirements.

[0115] Finally, in the computing node 51, the computing devices 511-518 are respectively deployed with the data processing stages S1, S2, S3, S4, S7, S10, S11, S12. Such deployment arrangement makes the sum of the to-be-transferred memory values of the first-type computing devices (511-514) corresponding to the first-type data processing stages be 5+4+3+2=14 GB, and the sum of the receivable memory values of the second-type computing devices (515-518) corresponding to the second-type data processing stages be 1+4+5+6=16 GB, satisfying the requirement that the sum of the to-be-transferred memory values of the first-type computing devices is less than the sum of the receivable memory values of the second-type computing devices, thereby guaranteeing the rationality and stability of the memory allocation of the computing node 51 in the preheating stage, and avoiding problems such as data processing failure or low efficiency caused by improper memory allocation.

[0116] In the computing node 52, the computing devices 521-528 are respectively deployed with the data processing stages S3, S5, S6, S8, S9, S13, S14, S15. For the computing node 52, the sum of the to-be-transferred memory values of the first-type computing devices (521-523) corresponding to the first-type data processing stages is 3+1+0=4 GB (the transferred memory value of the data processing stage S6 is 0), and the sum of the receivable memory values of the second-type computing devices (524-528) corresponding to the second-type data processing stages is 2+3+7+8+9=29 GB, also satisfying the above deployment rule.

[0117] In the entire data processing process, not only in the preheating stage, but also in the subsequent formal running stage, the memory usage of each data processing stage needs to be continuously monitored. Because with the continuous input of data and the advancement of processing logic, the memory demand of the data processing stage may change. For example, the data processing stage S1 may temporarily need an additional 2 GB of memory space for caching intermediate data when processing specific complex data. At this time, the system needs to have a dynamic adjustment mechanism. If there is available idle memory resource in the computing node 51, it can be preferentially allocated to the data processing stage S1, and part of the data in the data processing stage 51 is temporarily stored on other computing devices; if there is not enough idle memory, it may need to be allocated from other compressible memory data processing stages, or consider temporarily transferring part of the data to an external storage device, and then moving the data back to the memory after the data processing stage S1 processes the part of the data, so as to ensure that the data processing flow of the entire computing node 51 is not affected and can run efficiently.

[0118] For the computing node 52, if it is determined through pre-computation that the memory usage of the data processing stage S9 will sharply increase at a certain time, the system needs to evaluate the impact of this phenomenon on the memory balance of the entire computing node and take appropriate measures, such as optimizing the algorithm of the data processing stage S9 to reduce memory usage, or adjusting the distribution of other data processing stages on each computing device, and reasonably replacing some data processing stages with large acceptable memory values and low current memory usage with the data processing stage S9, in order to maintain the balance and stability of the memory allocation of the computing node 52, ensure the continuous and stable progress of the entire data processing task, and finally realize the accurate operation and effective output of the target model.

[0119] When the system needs to be expanded, new computing nodes are added, the data processing stages of the target model are updated, or the target model is upgraded or expanded in function, the newly added elements also need to be comprehensively analyzed and reasonably arranged according to this memory evaluation and allocation method, so as to ensure that the entire system always maintains high efficiency, stability and reliability in memory management, and can adapt to changing computing needs and data processing tasks. Moreover, as the data processing progresses and different batches of data flow in, the memory usage of each data processing stage may change, and a corresponding dynamic monitoring mechanism can be set up to keep track of these changes at all times. Once a trend that violates the above memory allocation rules is found, the distribution of each data processing stage on the computing device needs to be adjusted in time, or the memory usage of some data processing stages can be reduced by optimizing the relevant algorithm, to ensure that the entire computing system always runs stably and efficiently, and to ensure that the data processing task of the target model can be completed smoothly, providing reliable data support for the subsequent result output and use in actual application scenarios.

[0120] The above deployment process can be completed through pre-computation, or through actual operation of each data processing stage, sampling of running data, and completion based on measured data. The present disclosure does not make special limitations on this.

[0121] In addition to the above dynamic adjustment of the matching relationship between each computing device and data processing stage within the computing node, a fixed matching relationship of the data processing stage can also be formed in advance, thereby simplifying the deployment process.

[0122] In an exemplary embodiment, within the computing node, at least one computing device group that activates data transfer in a fixed matching relationship is determined according to the to-be-transferred memory value and the acceptable memory value, and the computing device group includes the first type of computing device and the second type of computing device.

[0123] For example, a fixed matching relationship is established between data processing stage S1 and data processing stage S16, and a fixed amount of activation data transferred by data processing stage S1 to data processing stage S16 is set (this can be memory usage x GB, or it can be determined based on the smallest indivisible unit of activation data). At least one group of data processing stages is formed according to the above logic, each group including at least one first-type data processing stage and at least one second-type data processing stage, and a fixed value is set for the activation data transferred by each first-type data processing stage to each second-type data processing stage during the warm-up phase.

[0124] In an exemplary embodiment, when forming a data processing stage group, the amount of active data transfer set within each data processing stage group can be minimized to reduce the pressure on the system caused by the active data transfer.

[0125] Next, based on the data processing stage group settings, select a data processing stage with the same number of computing devices as a computing node and deploy it in that computing node.

[0126] Figure 6 This is a schematic diagram of a computing device group in an exemplary embodiment of the present disclosure.

[0127] refer to Figure 6 ,by Figure 5 Taking the architecture shown as an example, assume that there are 6 data processing stage groups, namely (S1, S15, S16), (S2, S14), (S3, S4, S13), (S5, S11, S12), (S6, S10), and (S7, S8, S9). The three data processing stages (S1, S15, S16), (S2, S14), and (S3, S4, S13) can be set on computing node 51, and the three data processing stages (S5, S11, S12), (S6, S10), and (S7, S8, S9) can be set on computing node 52.

[0128] At computing node 51, the first type of computing devices corresponding to the first type of data processing stages S1~S4 are computing devices 511~514, and the second type of computing devices corresponding to the second type of data processing stages S13~S16 are computing devices 515~518. At computing node 52, the first type of computing devices corresponding to the first type of data processing stages S5~S7 are computing devices 521~523, and the second type of computing devices corresponding to the second type of data processing stages S8~S12 are computing devices 524~528. Each first type of computing device and each second type of computing device forms a computing device group according to the data processing stage group to which its corresponding data processing stage belongs. The active data transmission relationship within each computing device group is performed according to the fixed matching relationship set by the data processing stage group.

[0129] The above fixed matching relationship is only an example. A person skilled in the art can flexibly allocate according to the memory occupancy of each data processing stage in the warm-up stage, and adjust the correspondence between each computing node, computing device and data processing stage to realize the deployment of the large model.

[0130] After the target model is deployed, the operation of the target model formed based on the deployment method is introduced below.

[0131] Figure 7 is a flowchart of the data processing method in the embodiments of the present disclosure.

[0132] Reference Figure 7 , the data processing method 700 can include:

[0133] Step S71, in response to the training data input message, receiving the training data and processing;

[0134] Step S72, when it is determined to enter the warm-up stage, transferring the active data of the first type of data processing stage from the first type of computing device to the second type of computing device for temporary storage in one computing node, the first type of computing device being the computing device for deploying the first type of data processing stage, and the second type of computing device being the computing device for deploying the second type of data processing stage;

[0135] Step S73, in the backward propagation stage, taking the active data from the second type of computing device back to the first type of computing device.

[0136] In the exemplary embodiments, when the active data is formed in each first type of data processing stage, the active data can be transferred to the second type of computing device under the same computing node for temporary storage according to the pre-set transfer value. For example, the active data is transferred through the inter-GPU NVLink link. Since the data transfer between the computing devices in the same computing node can utilize the high-speed link in the computing node, the active data transfer can be realized quickly and with low power consumption, avoiding the burden of active data transfer on system operation.

[0137] In the exemplary embodiments, the active data formed by the first type of computing device can also be transferred to the second type of computing device having a fixed matching relationship with the first type of computing device for temporary storage. The fixed matching relationship is, for example, as shown in the embodiment. Figure 5

[0138] In addition to transferring the active data in the pre-set manner, in the embodiments of the present disclosure, the first type of computing device can also determine which second type of computing device or second type of computing devices to transfer the active data to and how much active data to transfer to each second type of computing device according to the measured memory of each computing device.

[0139] Figure 8 ​is a subflowchart of step S72 in one embodiment of the present disclosure.

[0140] Reference Figure 8 In an exemplary embodiment, step S72 can include:

[0141] Step S721, when determining to enter the preheating phase, determining the to-be-transferred memory value of each first-type computing device within the computing node;

[0142] Step S722, obtaining the receivable memory value of each second-type computing device within the computing node;

[0143] Step S723, determining the second-type computing device corresponding to each first-type computing device according to the to-be-transferred memory value and the receivable memory value;

[0144] Step S724, when the first-type computing device forms the activation data, transmitting the activation data corresponding to the to-be-transferred memory value of the memory occupancy number to the second-type computing device corresponding to the first-type computing device.

[0145] Figure 8 In the illustrated embodiment, the transfer direction and the transfer quantity of the activation data are determined in real time according to the actually measured to-be-transferred memory value of the first-type computing device and the receivable memory value of the second-type computing device, which can be flexibly adjusted according to the actual working condition, avoids the problem of memory overflow caused by the pre-set fixed matching relationship in actual operation, and enhances the system stability.

[0146] Exemplarily, the specific process of performing the activation data transmission can be as follows:

[0147] Determine the data quantity to be transferred: The data quantity to be transferred by the first-type computing device is usually the difference between the current saved mini-batch number and M(opt). This difference represents the memory quantity that can be released by the first-type computing device, and is also the memory quantity that can be increased by the second-type computing device.

[0148] Generate transfer scheduling: according to the role (first-type computing device or second-type computing device) of each GPU and the data quantity to be transferred, generate a transfer scheduling task.

[0149] After using the corresponding transmission decision to perform the calculation, the memory balancing target of all data processing stages is achieved.

[0150] Figure 9 is a schematic diagram of the model in which the method 700 is run. Figure 2 is a schematic diagram of the model in which the method 700 is run.

[0151] Reference Figure 9, the data processing stages S1-S4 are all deployed on the same computing node, or the data processing stages S1 and S4 are deployed on the same computing node, and the data processing stages S2 and S3 are deployed on the same computing node.

[0152] In the data processing stage S1, after the F 1,1 processing of the first micro-batch of data generates activation data, the activation data is transferred to the computing device where the data processing stage S4 is located through the inter-GPU high-speed data channel of the same computing node for temporary storage, and before the B 1,1 processing, the activation data is retrieved from the computing device where the data processing stage S4 is located for use. 4,1 Due to the preheating stage, the data processing stage S4 only needs to save the activation data of the F 1,1 , and the computing device where the data processing stage S4 is located has relatively sufficient memory and can temporarily store two copies of the activation data of the F 2,1 and the F 2,1 . Therefore, in the preheating stage, the memory occupancy of the computing devices corresponding to the data processing stages S1 and S4 is balanced, and the memory requirement of the computing device can be reduced.

[0153] Similarly, in the data processing stage S2, after the F 2,1 processing of the first micro-batch of data generates activation data, the activation data is transferred to the computing device where the data processing stage S3 is located through the inter-GPU high-speed data channel of the same computing node for temporary storage, and before the B 2,1 processing, the activation data is retrieved from the computing device where the data processing stage S3 is located for use.

[0154] In summary, the embodiments of the present disclosure divide the stages, transfer the activation data of the data processing stage with a higher sequence number to the computing device where the data processing stage with a lower sequence number is located for temporary storage during the forward propagation, load and retrieve the activation data from other computing devices for use during the backward propagation, cross-distribute the first type of data processing stage and the second type of data processing stage to the same computing node, use the inter-card NVLink high-bandwidth link to migrate the activation data, speed up the communication propagation, and flexibly adjust the transmission direction and the transmission amount of the activation data according to the load balancing of the second type of computing device as the receiver, balance the memory of each computing device as much as possible while reducing the memory pressure of each computing device.

[0155] The embodiments of the present disclosure balance the load of each computing device according to the memory utilization and other conditions of the computing node, can be applied in large-scale deep learning model training, federated learning, and other fields involving distributed training, and are also helpful for promoting the optimization of artificial intelligence acceleration cards and software and promoting the further development of artificial intelligence technology.

[0156] It should be noted that although several modules or units of the devices for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units for embodiment.

[0157] In the exemplary embodiments of the present disclosure, an electronic device capable of implementing the above-described method is also provided.

[0158] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method or a program product. Therefore, various aspects of the present disclosure can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0159] The electronic device 1000 according to this embodiment of the present disclosure will be described below with reference to Figure 10 Figure 10 The electronic device 1000 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0160] As shown in Figure 10 The components of the electronic device 1000 can include, but are not limited to, the at least one processing unit 1010 described above, the at least one storage unit 1020 described above, and a bus 1030 connecting different system components, including the storage unit 1020 and the processing unit 1010.

[0161] The storage unit stores program codes which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of the present specification. For example, the processing unit 1010 can perform the method as shown in the embodiments of the present disclosure.

[0162] The storage unit 1020 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 10201 and / or a cache memory unit 10202, and can further include a read-only memory (ROM) 10203.

[0163] ​The storage unit 1020 can also include a program / utility 10204 having a set of programs / modules 10205, each of which performs one or more tasks. Examples include an operating system, one or more application programs, other program modules, and program data, each or any combination thereof, which may

[0164] The bus 1030 can represent one or more of several types of bus structures, including a storage bus or bus controller, a peripheral bus, a graphics bus, a processor or local bus, using any of a variety of bus structures.

[0165] The electronic device 1000 can also communicate with one or more external devices 1100 such as a keyboard, a pointing device, a Bluetooth device, etc.; and one or more devices that enable a user to interact with the electronic device 1000. Additionally, the electronic device 1000 can communicate with one or more devices that enable the electronic device 1000 to communicate with one or more other computing devices. Such communication can be facilitated by an Input / Output (I / O) interface 1050. The electronic device 1000 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through a network adapter 1060. As depicted, the network adapter 1060 is in communication with the other modules of the electronic device 1000 through the bus 1030. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device 1000, including but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0166] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by software in combination with the necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (such as a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0167] In exemplary embodiments of the present disclosure, a computer readable storage medium having stored thereon a program product capable of implementing the above-described method of the specification is also provided. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present application.

[0168] The program product for implementing the above-described method according to the embodiments of the present application can take a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0169] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0170] The computer readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which readable program code is borne. Such propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit the program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] The program code contained on the readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0172] The program code may, for example, be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider.

[0173] Furthermore, the above-described diagrams are merely schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is readily understood that the processes shown in the above-described diagrams do not indicate or limit the time sequence of the processes. In addition, it is readily understood that the processes can be executed synchronously or asynchronously, for example, in a plurality of modules.

[0174] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

Claims

1. A model deployment method, characterized by, The method comprises: determining N data processing stages of a target model, the target model being divided into N data processing stages and running in a pipeline parallel manner; determining whether the data processing stage is a first type data processing stage or a second type data processing stage in a warm-up stage according to the stage number of the data processing stage, the first type data processing stage being an active data transfer side, the second type data processing stage being an active data receiving side, the memory requirement of the first type data processing stage in the warm-up stage being higher than the memory requirement of the second type data processing stage in the warm-up stage; deploying the first type data processing stage and the second type data processing stage in one computing node, the computing node comprising a plurality of computing devices, the plurality of computing devices comprising first type computing devices where the first type data processing stage is located and second type computing devices where the second type data processing stage is located; wherein the deploying the first type data processing stage and the second type data processing stage in one computing node comprises: determining the memory occupation value of the data processing stage in the warm-up stage according to the stage number of the data processing stage; obtaining the available memory of each computing device; determining the to-be-transferred memory value of the first type computing device corresponding to the first type data processing stage according to the difference between the memory occupation value of the first type data processing stage and the available memory of the computing device; determining the receivable memory value of the second type computing device corresponding to the second type data processing stage according to the difference between the available memory of the computing device and the memory occupation value of the second type data processing stage; setting the plurality of computing devices in one computing node as the first type computing devices and the second type computing devices respectively, deploying the first type data processing stage on the first type computing devices and deploying the second type data processing stage on the second type computing devices, the sum of the to-be-transferred memory values of the first type computing devices in the computing node being less than the sum of the receivable memory values of the second type computing devices.

2. The model deployment method of claim 1, wherein, The determining whether the data processing stage is a first type data processing stage or a second type data processing stage in a warm-up stage according to the stage number of the data processing stage comprises: when i<(N+1) / 2, the i-th data processing stage is the first type data processing stage; when i≥(N+1) / 2, the i-th data processing stage is the second type data processing stage; wherein, 1≤i≤N.

3. The model deployment method of claim 1, wherein, The deploying the first type data processing stage and the second type data processing stage in one computing node comprises: in the computing node, determining at least one computing device group for active data transfer in a fixed matching relationship according to the to-be-transferred memory value and the receivable memory value, the computing device group comprising the first type computing devices and the second type computing devices.

4. A data processing method, characterized by, executed by a target model deployed by the model deployment method of any one of claims 1-3, the data processing method comprising: in response to a training data input message, receiving training data and processing; transferring, in a computing node, activation data of a first type of data processing stage from a first type of computing device to a second type of computing device, the first type of computing device being a computing device deploying the first type of data processing stage, the second type of computing device being a computing device deploying a second type of data processing stage, when it is determined to enter a warm-up phase; fetching, in a backward propagation phase, the activation data from the second type of computing device back to the first type of computing device.

5. The data processing method of claim 4, wherein, The transferring, in a computing node, activation data of a first type of data processing stage from a first type of computing device to a second type of computing device, the first type of computing device being a computing device deploying the first type of data processing stage, the second type of computing device being a computing device deploying a second type of data processing stage, when it is determined to enter a warm-up phase, comprises: storing, in the first type of computing device, the activation data in a memory of the first type of computing device when the activation data is formed in the first type of computing device.

6. The data processing method of claim 4, wherein, The transferring, in a computing node, activation data of a first type of data processing stage from a first type of computing device to a second type of computing device, the first type of computing device being a computing device deploying the first type of data processing stage, the second type of computing device being a computing device deploying a second type of data processing stage, when it is determined to enter a warm-up phase, comprises: determining, when it is determined to enter a warm-up phase, a to-be-transferred memory value of each of the first type of computing devices in the computing node; obtaining a receivable memory value of each of the second type of computing devices in the computing node; determining, according to the to-be-transferred memory value and the receivable memory value, a corresponding second type of computing device for each of the first type of computing devices; transferring, in the first type of computing device, activation data whose memory occupancy corresponds to the to-be-transferred memory value to the corresponding second type of computing device when the activation data is formed in the first type of computing device.

7. The data processing method of claim 4, wherein, The transferring, in a computing node, activation data of a first type of data processing stage from a first type of computing device to a second type of computing device, the first type of computing device being a computing device deploying the first type of data processing stage, the second type of computing device being a computing device deploying a second type of data processing stage, when it is determined to enter a warm-up phase, comprises: transferring the activation data through an inter-GPU NVLink link.

8. An electronic device, comprising: comprises: a memory; and a processor coupled to the memory, the processor configured to perform the method of any one of claims 4-7 based on instructions stored in the memory.

9. A computer readable storage medium having stored thereon a program which, when executed by a processor, implements the method of any one of claims 4-7. The computer program which, when executed by a processor, implements the steps of the method of any one of claims 4-7.

10. A computer program product comprising a computer program, characterized in that, ​

Images