Model deployment method, data processing method and electronic equipment

Abstract

The invention provides a model deployment method, a data processing method and electronic equipment. The model deployment method comprises the steps that N data processing stages of a target model are determined, and the target model is divided into the N data processing stages to operate in a pipeline parallel mode; according to the stage sequence number of the data processing stage, it is determined that the data processing stage is a first-class data processing stage or a second-class data processing stage in a preheating stage, the first-class data processing stage is an activated data transfer party, and the second-class data processing stage is an activated data receiving party; the first type of data processing stage and the second type of data processing stage are deployed in a computing node, and the computing node comprises a plurality of computing devices. The plurality of computing devices comprise a first type of computing device where the first type of data processing stage is located and a second type of computing device where the second type of data processing stage is located. According to the embodiment of the invention, the deployment cost of large models can be reduced.

Description

Background Art

[0002] During the training process of a large model (LLM) that runs in a pipeline manner, the early warm-up stage needs to save all activation data during the forward propagation process for use in the backward propagation stage. This requires a large amount of memory and places high demands on the memory capacity of computing devices such as GPUs that run large models, resulting in high deployment costs for large models.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The disclosed embodiments are used to reduce the requirements for the memory capacity of hardware devices during the model training phase and reduce the cost of deploying large models.

[0005] According to a first aspect of an embodiment of the present disclosure, a model deployment method is provided, comprising: determining N data processing stages of a target model, wherein the target model is divided into N data processing stages and runs in a pipeline parallel manner; determining, according to the stage sequence number of the data processing stage, whether the data processing stage is a first type of data processing stage or a second type of data processing stage in a preheating stage, wherein the first type of data processing stage is an activated data transferor, and the second type of data processing stage is an activated data receiver; and deploying the first type of data processing stage and the second type of data processing stage in a computing node, wherein the computing node comprises a plurality of computing devices, and the plurality of computing devices comprises a first type of computing device where the first type of data processing stage is located, and a second type of computing device where the second type of data processing stage is located.

[0006] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing method, which is executed by a target model, and the target model is deployed by a model deployment method as described in any one of the above items. The data processing method comprises: responding to a training data input message, receiving and processing training data; when it is determined that a warm-up phase has been entered, transferring activation data of a first type of data processing phase from a first type of computing device to a second type of computing device for temporary storage within a computing node, the first type of computing device being a computing device for deploying the first type of data processing phase, and the second type of computing device being a computing device for deploying the second type of data processing phase; and in a backward propagation phase, retrieving the activation data from the second type of computing device back to the first type of computing device.

[0007] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the model deployment method as described in any one of the above items is implemented.

[0009] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0010] The disclosed embodiment determines the data processing stage with higher memory requirements in the preheating stage as the first type of data processing stage, and determines the data processing stage with relatively lower memory requirements in the preheating stage as the second type of data processing stage, and then deploys the first type of data processing stage and the second type of data processing stage in the same computing node. This allows the large model to transfer activation data from the first type of computing device where the first type of data processing stage is located to the second type of computing device where the second type of data processing stage is located in the same computing node during the preheating stage of the training process, thereby realizing memory sharing between different computing devices in the same computing node, thereby realizing memory transfer with low communication cost, reducing the memory requirements of the computing devices for the deployment of large models, and reducing the deployment cost of large models.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0013] Figure 1 is a schematic diagram of the data processing system that runs the target model.

[0014] Figure 2 It is a schematic diagram of the operating principle of the target model.

[0015] Figure 3 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0016] Figure 4 It is a sub-process of step S13 in one embodiment of the present disclosure.

[0017] Figure 5 yes Figure 4 A schematic diagram corresponding to the illustrated embodiment.

[0018] Figure 6 is a schematic diagram of a computing device group in an exemplary embodiment of the present disclosure.

[0019] Figure 7 It is a flow chart of the data processing method in an embodiment of the present disclosure.

[0020] Figure 8 It is a sub-flowchart of step S72 in one embodiment of the present disclosure.

[0021] Fig. 9 is method 700 in Figure 2 Schematic diagram of the model run for the example shown.

[0022] Fig.10 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0024] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0025] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0026] To explain the principle, let's first introduce the hardware system for deploying large models.

[0027] Figure 1 is a schematic diagram of the data processing system that runs the target model.

[0028] refer to Figure 1 In the embodiment of the present disclosure, the data processing system 100 may include: At least one computing node 1, each computing node 1 includes multiple computing devices 11 and at least one central processing unit 12, the central processing unit 12 interacts with the multiple computing devices 11, and data exchange is achieved between the multiple computing devices 11 through a high-speed data transmission channel.

[0029] Each computing device 11 is provided with a data processing stage Si of the target model, the target model is a large model (LLM), and the target model is divided into N data processing stages for operation in a pipeline parallel manner, 1≤i≤N.

[0030] The computing node 1 includes but is not limited to a server and a single workstation in a workstation cluster. The computing device 11 is, for example, a GPU (Graphics Processing Unit) or an application-specific integrated circuit (ASIC), such as a TPU (Tensor Processing Unit), which is arranged in the computing node 1. Alternatively, the computing device 11 may also be implemented by a field programmable gate array (FPGA).

[0031] During the operation of the target model, data flows in turn in the stages that each computing device is responsible for. After the previous computing device completes the task of its corresponding data processing stage Si, it will quickly pass the processing result to the next computing device corresponding to the stage Si+1 for the next step of processing. In the same computing node 1, the various computing devices 11 are calculated through a high-speed data transmission channel, such as an NVLink link. Between different computing nodes 1, the computing devices 11 interact with each other through a high-speed network interconnection architecture, such as using InfiniBand network or high-speed Ethernet technology. Among them, NVLink is a high-speed, high-bandwidth interconnection technology developed by NVIDIA. It is mainly used to connect multiple GPUs and realize direct communication between GPUs. Its purpose is to accelerate the transmission of data between multiple GPUs and overcome the problem of insufficient bandwidth of the traditional PCI-Express (PCI-e) bus in multi-GPU communication scenarios.

[0032] Therefore, the data transmission speed between computing devices 11 within the same computing node 1 is significantly higher than the data transmission speed between computing devices 11 between different computing nodes 1 .

[0033] In an exemplary embodiment, the data processing system 100 may include a memory management module. The memory management module is responsible for reasonably allocating and scheduling memory resources of each computing node and each computing device to meet the needs of large models for data storage and reading at different data processing stages. The memory management module can be set on the computing device 11 of a computing node 1, or it can exist in the data processing system 100 as a dedicated control unit independent of each computing device 11 to control data transmission between multiple computing nodes 1.

[0034] The data processing method provided by the embodiment of the present disclosure can be implemented by a memory management module, and can also be implemented by a target model by setting the operation mode of the target model.

[0035] Figure 2 It is a schematic diagram of the operating principle of the target model.

[0036] refer to Figure 2 , the target model is divided into N (in Figure 2 In the embodiment, N=4) data processing stages are operated in a pipeline parallel manner and a 1F1B (One-Forward-One-Backward, one forward propagation-one backward propagation strategy) mode.

[0037] In the forward propagation phase, data starts from the first data processing stage S1 and flows through each subsequent stage in sequence. Data processing stage S1 generally performs preliminary feature extraction and conversion on the input data, such as word vector encoding of text data or basic feature extraction of image data, and then passes the processed intermediate data to data processing stage S2. Data processing stage S2 further processes the results of data processing stage S1, which may involve more complex feature fusion or semantic understanding-related calculations, and so on, until the data completes the final forward processing step in data processing stage SN and obtains the forward output result of the model.

[0038] refer to Figure 2 As shown, Figure 2 A square in the middle represents the processing of a micro-batch of data in a data processing stage. F represents the processing in the forward propagation stage, and B represents the processing in the backward propagation stage. The first value in the subscript represents the data processing stage, and the second value represents the batch of data, such as F 2,1 represents the second data processing stage S2 that processes the second batch of data in the forward propagation stage, which can also be called the forward propagation processing of the second batch of data to the second data processing stage S2; correspondingly, B 2,1It represents the second data processing stage S2 that processes the second batch of data in the back propagation stage, which can also be called the back propagation processing of the second batch of data to the second data processing stage S2. The process of the first batch of data being completely processed is F 1,1 →F 2,1 →F 3,1 →F 4,1 →B 4,1 →B 3,1 →B 2,1 →B 1,1 ,like Figure 2 As shown by the arrow in the middle. The processing flow of other batches of data is the same and will not be repeated here.

[0039] In the forward propagation phase of the model, the intermediate results calculated at each data processing stage are called activation data. These activation data contain the feature information gradually extracted by the model in the process of processing the input data. For example, in a simple multi-layer neural network, after the input layer receives the data, it is multiplied by the weights of the first layer of neurons and added with the bias, and then the output result obtained by the activation function is the activation data of the first layer. This data will continue to participate in the calculation as the input of the next layer. When training the model, the activation data needs to be saved at each data processing stage in the forward propagation phase to complete the model training later.

[0040] After completing a forward propagation, the backward propagation will be carried out immediately, and the backward propagation starts from the data processing stage SN. The data processing stage SN calculates its own gradient with respect to the loss function and passes it to the data processing stage SN-1. The data processing stage SN-1 uses the received gradient information combined with its own calculation logic (model parameters) in the forward propagation and the saved activation data to calculate its own gradient and continue to pass it back to the data processing stage SN-2. In this process, each stage updates its own model parameters to optimize the model, so that the model can gradually learn the rules and characteristics in the data.

[0041] exist Figure 2 In the example, the data processing stage S2 is in F 2,1 When F 2,1 activation data, so that it can be used in the subsequent B 2,1 When using this F 2,1 The activation data updates the parameters of the data processing stage S2. For the data processing stage S2, after completing B 2,1 Before, you need to temporarily save F 2,1 activation data.

[0042] Therefore, during the operation of the 1F1B mode, there are a warm-up phase (Warm-up Phase), a steady-state phase (Steady-state Stage), and a cool-down phase (Cool-down Phase). In the warm-up phase, each data processing stage only performs forward propagation processing, without backward propagation processing, and the generated activation data continues to increase, and the memory pressure in each computing device 11 increases; in the steady-state phase, the data processing stage with a later sequence number begins to perform backward propagation processing, uses the activation data, and then deletes the activation data. At the same time, other data processing stages are still generating new activation data, and the memory pressure of each computing device remains basically unchanged; in the cool-down phase, each data processing stage has completed the forward propagation processing and only performs backward propagation processing at the beginning, continuously consuming the temporarily stored activation data, so that the memory pressure of each computing device gradually decreases. After the cool-down phase, the accumulated gradient of each micro-batch is used to update the model parameters of each data processing stage.

[0043] observe Figure 2 It can be seen that the data processing stage S1 must store the activation data of microbatches as many as the Pipeline parallelism during the warm-up stage. In other data processing stages, the number of microbatches processed in the warm-up stage decreases linearly, and the last data processing stage S4 only keeps the activation data of one microbatch. Therefore, the imbalance of memory usage exists in the entire Pipeline stage (S1~S4), and the degree of imbalance is amplified as the Pipeline parallelism is upgraded, that is, the number of data processing stages increases.

[0044] Among them, the memory occupied by the data processing stage Si on the computing device is M(i)=W(i)+A(i), W(i) represents the size of the model parameters including the optimizer state, which is a constant value W0; A(i) represents the memory usage value of the saved activation data, , MB_num(i) is the number of micro-batches that need to save activation parameters, and A0 is the memory usage value of the activation data corresponding to each micro-batch data. When each data processing stage has the same number of layers, A0 is a constant and M(i) depends on the number of micro-batches MB_num(i).

[0045] Therefore, due to the high memory pressure in the warm-up phase, large models have high memory requirements for computing devices, which puts pressure on the deployment cost of large models.

[0046] To this end, the embodiment of the present disclosure solves this problem through a model deployment method 300.

[0047] Figure 3 is a flowchart of a model deployment method in an exemplary embodiment of the present disclosure.

[0048] refer to Figure 3 , the model deployment method 300 may include: Step S11, determining N data processing stages of the target model, wherein the target model is divided into N data processing stages and runs in a pipeline parallel manner; Step S12, determining, according to the stage sequence number of the data processing stage, whether the data processing stage is a first type of data processing stage or a second type of data processing stage in the preheating stage, the first type of data processing stage is to activate the data transfer side, and the second type of data processing stage is to activate the data receiver side; Step S13, deploying the first type of data processing stage and the second type of data processing stage in a computing node, the computing node includes multiple computing devices, and the multiple computing devices include a first type of computing device where the first type of data processing stage is located and a second type of computing device where the second type of data processing stage is located.

[0049] The disclosed embodiment determines the data processing stage with higher memory requirements in the preheating stage as the first type of data processing stage, and determines the data processing stage with relatively lower memory requirements in the preheating stage as the second type of data processing stage, and then deploys the first type of data processing stage and the second type of data processing stage in the same computing node. This allows the large model to transfer activation data from the first type of computing device where the first type of data processing stage is located to the second type of computing device where the second type of data processing stage is located in the same computing node during the preheating stage of the training process, thereby realizing memory sharing between different computing devices in the same computing node, thereby realizing memory transfer with low communication cost, reducing the memory requirements of the computing devices for the deployment of large models, and reducing the deployment cost of large models.

[0050] The method 300 is described in detail below.

[0051] In step S11, N data processing stages of the target model are determined.

[0052] Determining N data processing stages is mainly used to determine the number N of data processing stages and the processing order corresponding to each data processing stage, and then setting serial numbers for the data processing stages, setting the data processing stage with processing order i as data processing stage Si, 1≤i≤N.

[0053] In step S12, it is determined according to the stage sequence number of the data processing stage whether the data processing stage is a first type of data processing stage or a second type of data processing stage in the preheating stage, the first type of data processing stage is to activate the data transfer side, and the second type of data processing stage is to activate the data receiver side.

[0054] like Figure 2As shown, when the number of micro-batches m processed by the data processing stage Si is greater than the number of data processing stages N, for example Figure 2 F 1,5 ,have: (1) (2) The memory decreases linearly with the increase of the sequence number of the data processing stage. When i≤N / 2, it occupies more memory, and the memory occupies less in the subsequent stages. That is, in the first half of the data processing stage, since there are more micro-batch activation data that need to be saved to wait for backpropagation, the memory requirement is high; for the second half of the data processing stage, since backpropagation has begun and part of the activation data has been consumed, the memory requirement begins to decrease.

[0055] Therefore, in an exemplary embodiment, step S12 may include: When i<(N+1) / 2, the i-th data processing stage is the first type of data processing stage; When i ≥ (N+1) / 2, the i-th data processing stage is the second type of data processing stage; Among them, N≥1, 1≤i≤N.

[0056] Among them, the first type of data processing stage is the activation data transferor, which is configured to transfer the generated activation data to reduce the memory pressure on the computing device 11 where it is located; the second type of data processing stage is the activation data receiver, which is configured to receive and temporarily store the activation data from the first type of data processing stage, and allow the first type of data processing stage to retrieve the activation data when it needs to use the activation data in the backward propagation stage, thereby utilizing the idle memory capacity to share the memory pressure from other computing devices 11, solving the problem of high memory pressure and high requirements on the computing devices 11 caused by the warm-up stage, reducing the memory requirements for each computing device 11, and reducing the deployment cost of large models.

[0057] The above example can be briefly described as balancing the memory occupied by the activation data of MB_num(i) micro-batches and the activation data of MB_num(Ni-1) micro-batches by cross-distributing the data processing stages before and after N / 2 to the same node for activation transmission.

[0058] So that: (3) The sum of the maximum number of mini-batches saved during the matching phase is: (4) In the stable phase, it is N+2, where the optimally balanced number of micro-batches is: (5) The optimal memory balance is: (6) Select the data processing stage in which the number of micro-batches MB_num(i) saved in all data processing stages is less than or equal to the optimal number of micro-batches MB_num(opt) as the first type of data processing stage, and send the activation data of at most (Ni)-MB_num(opt)+1 micro-batches to the computing device where the second type of data processing stage is located. In this way, a pair of first type computing devices and second type computing devices with an activation data transfer relationship can save at most i+1 micro-batch data activation data, which can be expressed as, when MB_num(pi-1)≤MB_num(opt), there is: (7) The above embodiment makes an exemplary division according to the first half of the data processing stage / the second half of the data processing stage. In other embodiments, it is also possible to determine which data processing stages are the first type of data processing stages, which data processing stages are the second type of data processing stages, and which data processing stages are neither the first type of data processing stages nor the second type of data processing stages based on the memory usage values ​​of each data processing stage in the preheating stage obtained by actual calculation.

[0059] For example, let N=8, and the memory occupancy values ​​of the 1st to 8th data processing stages in the warm-up stage are 100, 95, 90, 85, 80, 75, 70, and 65 storage units (the data is only an example, and the storage unit is, for example, GB), and each data processing stage corresponds to a computing device, and the available memory of each computing device is 90 (also, for example, GB). At this time, the 1st and 2nd data processing stages can be set as the first type of data processing stages, and the 7th and 8th data processing stages can be set as the second type of data processing stages. In the warm-up stage, the 1st data processing stage transfers 10 activation data corresponding to the memory occupancy values ​​to the computing device where the 8th data processing stage is located, and the 2nd data processing stage transfers 5 activation data corresponding to the memory occupancy values ​​to the computing device where the 7th data processing stage is located. At the same time, no special settings are made for the 3rd, 4th, 5th, and 6th data processing stages to reduce the transfer amount of activation data and reduce the system operation pressure.

[0060] In the above example, the first and second data processing stages can also be set as the first type of data processing stages, and the eighth data processing stage can be set as the second type of data processing stage. Thus, in the warm-up stage, the first data processing stage can be set to transfer activation data corresponding to 10 memory occupancy values ​​to the computing device where the eighth data processing stage is located, and the second data processing stage can be set to transfer activation data corresponding to 5 memory occupancy values ​​to the computing device where the eighth data processing stage is located, thereby reducing the number of computing devices involved in the transfer of activation data and reducing the system operating pressure.

[0061] That is, in the embodiments of the present disclosure, each data processing stage can be set as a first-category data processing stage, can be set as a second-category data processing stage, or can be set without special settings. The number of first-category data processing stages and the number of second-category data processing stages can be the same or different. That is, the number of first-category data processing stages can be set to be greater than the number of second-category data processing stages, and the number of first-category data processing stages can also be set to be less than the number of second-category data processing stages. For example, the activation data of a first-category data processing stage can be transferred to two or more second-category data processing stages for temporary storage, so as to further reduce the requirements for the memory capacity of each computing device.

[0062] In other embodiments, the requirement for available memory of the computing device can be determined by setting a first data processing stage and a second data processing stage according to the memory occupancy value of each data processing stage in the preheating stage, thereby further reducing the hardware cost of the computing device.

[0063] Taking the above example, the 1st to 3rd data processing stages can be set as the first type of data processing stages, and the 6th to 8th data processing stages can be set as the second type of data processing stages. In the warm-up stage, the 1st data processing stage transfers activation data of 20 memory occupancy values ​​to the computing device where the 8th data processing stage is located, the 2nd data processing stage transfers activation data of 15 memory occupancy values ​​to the computing device where the 7th data processing stage is located, and the 3rd data processing stage transfers activation data of 5 memory occupancy values ​​to the computing device where the 6th data processing stage is located. Therefore, after the transfer, the maximum memory occupancy value of the computing device pairs where each data processing stage is located is 85, and then the available memory of each computing device is set to 85, and the computing devices are deployed according to the available memory of 85 to reduce the deployment cost of large models.

[0064] There are many ways to set the first data processing stage and the second data processing stage. For example, the number of activated data to be transferred can be determined based on the smallest indivisible unit of the activated data, thereby determining the first data processing stage and the second data processing stage. The present disclosure does not impose any special restrictions on this.

[0065] By transferring the activation data generated by the first data processing stage in the warm-up phase to the computing device where the second data processing stage is located, the memory requirements of the target model on the computing device during the warm-up phase can be reduced, thereby reducing the deployment cost of large models.

[0066] Since data needs to be transferred between computing devices, in order to overcome the impact of the additional data transfer action on efficiency, the embodiments of the present disclosure adjust the correspondence between the computing devices and the data processing stages, so as to avoid the related art of setting the computing devices according to the sequence number of the data processing stage, resulting in low data transfer efficiency and reduced overall system operation efficiency.

[0067] In step S13, the first type of data processing stage and the second type of data processing stage are deployed in a computing node, and the computing node includes multiple computing devices, and the multiple computing devices include a first type of computing device where the first type of data processing stage is located and a second type of computing device where the second type of data processing stage is located.

[0068] In the embodiment of the present disclosure, each data processing stage is deployed on each computing device of the computing node according to the data transfer relationship rather than the sequence number of the data processing stage.

[0069] In an exemplary embodiment, half of the computing devices in a computing node can be directly configured to deploy the first type of data processing stage, and the other half of the computing devices can be configured to deploy the second type of data processing stage. In other embodiments, more detailed settings can be made to better achieve the purpose of reducing the deployment cost of large models.

[0070] Figure 4 It is a sub-process of step S13 in one embodiment of the present disclosure.

[0071] refer to Figure 4 , step S13 may include: Step S131, determining the memory usage value of the data processing stage in the preheating stage according to the stage sequence number of the data processing stage; Step S132, obtaining available memory of each computing device; Step S133, determining the to-be-transferred memory value of the first type of computing device corresponding to the first type of data processing stage according to the difference between the memory usage value of the first type of data processing stage and the available memory of the computing device; Step S134, determining the receivable memory value of the second type of computing device corresponding to the second type of data processing stage according to the difference between the available memory of the computing device and the memory occupancy value of the second type of data processing stage; Step S135, respectively set multiple computing devices in a computing node as first-category computing devices and second-category computing devices, deploy a first-category data processing stage on the first-category computing devices, and deploy a second-category data processing stage on the second-category computing devices, and the sum of the memory values ​​to be transferred corresponding to the first-category computing devices in the computing node is less than the sum of the receivable memory values ​​corresponding to the second-category computing devices.

[0072] exist Figure 4 In the embodiment, the memory usage value of each data processing stage in the warm-up stage (such as the example given above) and the available memory of each computing device can be determined first, and then, under the current hardware conditions (the available memory limit of the computing device), it is determined which data processing stages' activation data need to be transferred out, and how much activation data to transfer. In addition, it is necessary to find a computing device for temporarily storing the transferred activation data, so it is necessary to determine which computing devices in which the data processing stages are located can accept the transferred activation data, and calculate how much activation data these computing devices can accept at most.

[0073] In an exemplary embodiment, first, each data processing stage can be set as a first type of data processing stage (memory occupancy value is greater than the available memory) and a second type of data processing stage (memory occupancy value is less than the available memory) according to whether the memory occupancy value of the data processing stage in the preheating stage is greater than the available memory of the computing device. Next, according to the gap between the memory occupancy value and the available memory, it is determined how much activation data the first type of data processing stage needs to transfer out in the preheating stage (according to the memory occupancy value, referred to as the transfer memory value) and how much activation data the second type of data processing stage can accept in the preheating stage (according to the memory occupancy value, referred to as the receivable memory value). Finally, when deploying the first type of data processing stage and the second type of data processing stage on the computing device of the same computing node, it is necessary to ensure that the sum of the memory values ​​to be transferred corresponding to the first type of computing devices in each computing node is less than the sum of the receivable memory values ​​corresponding to the second type of computing devices.

[0074] Figure 5 yes Figure 4 A schematic diagram corresponding to the illustrated embodiment.

[0075] refer to Figure 5, there are computing nodes 51 and 52, each of which is equipped with 8 computing devices, namely computing devices 511~518 and computing devices 521~528, and the available memory value of each computing device is 10GB. The target model has a total of 16 data processing stages, namely data processing stages S1~S16. The memory value occupied by each data processing stage in the warm-up stage decreases in sequence, namely 15GB, 14GB, 13GB, 12GB, 11GB, 10GB, 9GB, 8GB, 7GB, 6GB, 5GB, 4GB, 3GB, 2GB, 1GB, 0.5GB (the data is only for example and has no practical significance).

[0076] Therefore, the memory usage of data processing stages S1-S5 is greater than 10GB, which belongs to the first type of data processing stage. The transfer memory value of data processing stage S1 is 15-10=5GB, the transfer memory value of data processing stage S2 is 14-10=4GB, the transfer memory value of data processing stage S3 is 13-10=3GB, the transfer memory value of data processing stage S4 is 12-10=2GB, and the transfer memory value of data processing stage S5 is 11-10=1GB.

[0077] The memory usage of data processing stages S7-S16 is less than 10GB, which belongs to the second type of data processing stage. The receivable memory value of data processing stage S7 is 10-9=1GB; The receivable memory value of data processing stage S8 is 10-8=2GB; The receivable memory value of data processing stage S9 is 10-7=3GB; The receivable memory value of data processing stage S10 is 10-6=4GB; The receivable memory value of data processing stage S11 is 10-5=5GB; The receivable memory value of the data processing stage S12 is 10-4=6GB; The receivable memory value in the data processing stage S13 is 10-3=7GB; The receivable memory value in the data processing stage S14 is 10-2=8GB; The receivable memory value in the data processing stage S15 is 10-1=9GB; The receivable memory value of the data processing stage S16 is 10-0.5=9.5GB.

[0078] In computing node 51, if computing device 511 deploys S1, its memory value to be transferred is 5GB; computing device 512 deploys S7, and its receivable memory value is 1GB. To ensure that the sum of the memory values ​​to be transferred corresponding to the first type of computing devices in computing node 51 is less than the sum of the receivable memory values ​​corresponding to the second type of computing devices, it is necessary to reasonably arrange the deployment of other data processing stages on the remaining computing devices. For example, if S2 is deployed on computing device 513, and its memory value to be transferred is 4GB, then it is necessary to deploy enough second-type data processing stages with larger receivable memory values ​​on subsequent computing devices, such as deploying S10 on computing device 514, and its receivable memory value is 4GB, etc. By continuously adjusting the combination, the memory allocation of the entire computing node 51 is balanced.

[0079] Similarly, in computing node 52, deployment planning is performed according to the memory usage characteristics of each data processing stage. For example, if computing device 521 deploys S3, its memory value to be transferred is 3GB, then when computing device 522 deploys S8, its receivable memory value is 2GB, and the subsequent deployment on computing devices must continue to be comprehensively considered to meet the memory allocation requirements.

[0080] Finally, in computing node 51, computing devices 511~518 respectively deploy data processing stages S1, S2, S3, S4, S7, S10, S11, and S12. Such deployment arrangement makes the sum of the memory values ​​to be transferred corresponding to the first type of computing devices (511-514) where the first type of data processing stage is located 5+4+3+2=14GB, and the sum of the receivable memory values ​​corresponding to the second type of computing devices (515-518) where the second type of data processing stage is located is 1+4+5+6=16GB, which meets the requirement that the sum of the memory values ​​to be transferred corresponding to the first type of computing devices is less than the sum of the receivable memory values ​​corresponding to the second type of computing devices, thereby ensuring the rationality and stability of memory allocation of computing node 51 in the warm-up stage, and avoiding problems such as data processing failures or inefficiency caused by improper memory allocation.

[0081] In computing node 52, computing devices 521-528 respectively deploy data processing stages S3, S5, S6, S8, S9, S13, S14, and S15. For computing node 52, the sum of the memory values ​​to be transferred corresponding to the first type of computing devices (521-523) where the first type of data processing stage is located is 3+1+0=4GB (the transfer memory value of data processing stage S6 is 0), and the sum of the receivable memory values ​​corresponding to the second type of computing devices (524-528) where the second type of data processing stage is located is 2+3+7+8+9=29GB, which also complies with the above deployment rules.

[0082] During the entire data processing process, not only should we pay attention to memory allocation in the warm-up phase, but we also need to continuously monitor the memory usage of each data processing phase in the subsequent formal operation phase. Because with the continuous input of data and the advancement of processing logic, the memory requirements of the data processing phase may change. For example, when processing specific complex data, the data processing stage S1 may temporarily need an additional 2GB of memory space to cache intermediate data. At this time, the system needs to have a dynamic adjustment mechanism. If there are still available free memory resources in the computing node 51, they can be allocated to the data processing stage S1 first, and part of the data in the data processing stage 51 can be transferred to other computing devices for temporary storage; if there is not enough free memory, it may be necessary to allocate from other data processing stages with compressible memory, or consider temporarily transferring part of the data to an external storage device, and move the data back to the memory after S1 has processed the part of the data, so as to ensure that the data processing flow of the entire computing node 51 is not affected and can run efficiently.

[0083] The same is true for computing node 52. If it is determined through pre-calculation that the memory usage of data processing stage S9 will increase sharply at a certain moment, the system needs to evaluate the impact of this phenomenon on the memory balance of the entire computing node and take corresponding countermeasures, such as optimizing the algorithm of data processing stage S9 to reduce memory usage, or adjusting the distribution of other data processing stages on various computing devices, and reasonably replacing some data processing stages with large receivable memory values ​​and low current memory usage with data processing stage S9, so as to maintain the balance and stability of memory allocation of computing node 52, ensure that the entire data processing task can be continuously and stably promoted, and ultimately achieve accurate operation and effective output of the target model.

[0084] When the system needs to be expanded, new computing nodes are added, the data processing stage of the target model is updated, or the target model is upgraded or its functions are expanded, the newly added elements should be comprehensively analyzed and reasonably arranged according to this set of memory evaluation and allocation methods, so as to ensure that the entire system always maintains high efficiency, stability and reliability in memory management, and can adapt to the ever-changing computing needs and data processing tasks. In addition, with the advancement of data processing and the influx of different batches of data, the memory usage of each data processing stage may change, and a corresponding dynamic monitoring mechanism can be set to keep an eye on these changes. Once a trend that violates the above memory allocation rules is found, it is necessary to adjust the distribution of each data processing stage on the computing device in a timely manner, or reduce the memory usage of certain data processing stages by optimizing related algorithms, to ensure that the entire computing system can always run stably and efficiently, to ensure that the data processing tasks of the target model can be successfully completed, and to provide reliable data support for subsequent result output and use in actual application scenarios.

[0085] The above deployment process can be completed by pre-calculation or by actually running each data processing stage, sampling the running data process, and completing it according to the measured data. The present disclosure does not impose any special restrictions on this.

[0086] In addition to the above-mentioned dynamic adjustment of the matching relationship between each computing device within the computing node and the data processing stage, a fixed matching relationship of the data processing stage can also be formed in advance to simplify the deployment process.

[0087] In an exemplary embodiment, within a computing node, at least one computing device group for activating data transfer with a fixed matching relationship is determined based on the memory value to be transferred and the receivable memory value, and the computing device group includes a first type of computing device and a second type of computing device.

[0088] For example, a fixed matching relationship is set between the data processing stage S1 and the data processing stage S16, and a fixed amount of activation data transferred from the data processing stage S1 to the data processing stage S16 is set (which can be the memory usage xGB, or can be determined according to the smallest indivisible unit of activation data). At least one data processing stage group is formed according to the above logic, each group includes at least one first-type data processing stage and at least one second-type data processing stage, and a fixed value of activation data transferred from each first-type data processing stage to each second-type data processing stage during the warm-up phase is set.

[0089] In an exemplary embodiment, when forming the data processing stage groups, it may be considered that the activation data transfer amount set in each data processing stage group is as small as possible to reduce the pressure on the system caused by the activation data transfer.

[0090] Next, according to the setting of the data processing stage group, data processing stages equal in number to the number of computing devices in a computing node are selected and arranged in the computing node.

[0091] Figure 6 is a schematic diagram of a computing device group in an exemplary embodiment of the present disclosure.

[0092] refer to Figure 6 ,by Figure 5 Taking the shown architecture as an example, assuming that six data processing stage groups are formed, namely (S1, S15, S16), (S2, S14), (S3, S4, S13), (S5, S11, S12), (S6, S10), (S7, S8, S9), the three groups of data processing stages (S1, S15, S16), (S2, S14), (S3, S4, S13) can be set on computing node 51, and the three groups of data processing stages (S5, S11, S12), (S6, S10), (S7, S8, S9) can be set on computing node 52.

[0093] In computing node 51, the first computing devices corresponding to the first data processing stages S1 to S4 are computing devices 511 to 514, and the second computing devices corresponding to the second data processing stages S13 to S16 are computing devices 515 to 518; in computing node 52, the first computing devices corresponding to the first data processing stages S5 to S7 are computing devices 521 to 523, and the second computing devices corresponding to the second data processing stages S8 to S12 are computing devices 524 to 528. Each first computing device and second computing device forms a computing device group according to the data processing stage group to which the corresponding data processing stage belongs, and the activation data transmission relationship within each computing device group is performed according to the fixed matching relationship set by the data processing stage group.

[0094] The above fixed matching relationship is only an example. Those skilled in the art can flexibly allocate it according to the memory usage of each data processing stage in the warm-up stage, and adjust the corresponding relationship between each computing node, computing device and data processing stage to realize the deployment of large models.

[0095] After the target model is deployed, the operation of the target model formed based on the deployment method is described below.

[0096] Figure 7 It is a flow chart of the data processing method in an embodiment of the present disclosure.

[0097] refer to Figure 7 , the data processing method 700 may include: Step S71, responding to the training data input message, receiving and processing the training data; Step S72, when it is determined that the warm-up phase has begun, in a computing node, the activation data of the first data processing phase is transferred from the first computing device to the second computing device for temporary storage, the first computing device being the computing device for deploying the first data processing phase, and the second computing device being the computing device for deploying the second data processing phase; Step S73, in the back propagation phase, the activation data is retrieved from the second type of computing device to the first type of computing device.

[0098] In an exemplary embodiment, when activation data is generated in each first-type data processing stage, it can be transferred to the second-type computing device under the same computing node for temporary storage according to a preset transfer value, and the transfer method is, for example, to transmit the activation data through an NVLink link between GPUs. Since data transfer between computing devices within the same computing node can utilize the high-speed link within the computing node, fast and low-power activation data transfer can be achieved, avoiding the burden of activation data transfer on system operation.

[0099] In an exemplary embodiment, when the first type of computing device generates activation data, the activation data may be transmitted to a second type of computing device having a fixed matching relationship with the first type of computing device for temporary storage. Figure 5 As shown in the embodiment shown.

[0100] In addition to transferring activation data in a pre-set manner, in the embodiments of the present disclosure, it is also possible to determine which second-category computing devices the first-category computing device will transfer the activation data to and how much activation data to transfer to each second-category computing device based on the measured memory of each computing device.

[0101] Figure 8 It is a sub-flowchart of step S72 in one embodiment of the present disclosure.

[0102] refer to Figure 8 In an exemplary embodiment, step S72 may include: Step S721, when it is determined that the warm-up phase has begun, determining the memory value to be transferred of each first-category computing device in the computing node; Step S722, obtaining the receivable memory value of each second-category computing device in the computing node; Step S723, determining the second type of computing device corresponding to each first type of computing device according to the memory value to be transferred and the receivable memory value; Step S724, when the first type of computing device forms activation data, the activation data of the memory occupancy number corresponding to the memory value to be transferred is transmitted to the second type of computing device corresponding to the first type of computing device.

[0103] Figure 8 In the illustrated embodiment, the transfer direction and transfer quantity of the activation data are determined in real time based on the actually measured memory values ​​to be transferred of the first category computing devices and the receivable memory values ​​of the second category computing devices, and can be flexibly adjusted according to the actual working conditions to avoid problems in the pre-set fixed matching relationship during actual operation, resulting in memory overflow, thereby enhancing system stability.

[0104] Exemplarily, the specific process of activating data transmission may be as follows: Determine the amount of data to be transferred: The amount of data that the first computing device needs to transfer is usually the difference between its current number of stored micro-batches and M(opt). This difference represents the amount of memory that the first computing device can release and the amount of memory that the second computing device can increase.

[0105] Generate transfer schedule: Generate transfer schedule tasks according to the role of each GPU (first-class computing device or second-class computing device) and the amount of data to be transferred.

[0106] After calculation with the corresponding transfer decisions, the memory balance goal of all data processing stages is achieved.

[0107] Fig. 9 is method 700 in Figure 2 Schematic diagram of the model run for the example shown.

[0108] refer to Fig. 9 , it is assumed that data processing stages S1~S4 are deployed on the same computing node, or data processing stages S1 and S4 are deployed on the same computing node, and data processing stages S2 and S3 are deployed on the same computing node.

[0109] In the data processing stage S1, the first micro-batch data is processed by F 1,1 After the activation data is generated, the activation data is transferred to the computing device where the data processing stage S4 is located through the high-speed data path between GPUs of the same computing node for temporary storage. 1,1 Before that, the activation data is retrieved from the computing device where the data processing stage S4 is located. Due to the warm-up phase, the data processing stage S4 only needs to save F 4,1 The activation data of the computing device has sufficient memory to temporarily store the F from the data processing stage S1. 1,1 and F 2,1 Therefore, in the warm-up phase, the memory usage of the computing devices corresponding to the data processing phases S1 and S4 is relatively balanced, which can reduce the memory requirements for the computing devices.

[0110] Similarly, in the data processing stage S2, the first micro-batch data is subjected to F 2,1 After the activation data is generated, the activation data is transferred to the computing device where the data processing stage S3 is located through the high-speed data path between GPUs of the same computing node for temporary storage. 2,1 Previously, the activation data was retrieved from the computing device where the data processing stage S3 was located for use.

[0111] To summarize, the disclosed embodiment divides the stages, transfers the activation data of the data processing stage with an earlier sequence number to the computing device where the data processing stage with a later sequence number is located for temporary storage during forward propagation, and loads and retrieves the activation data from other computing devices for use during backward propagation. The first type of data processing stage and the second type of data processing stage are cross-distributed to the same computing node, and the activation data can be migrated by using the NVLink high-bandwidth link between cards, which can speed up the communication transmission speed. The transmission direction and amount of the activation data can also be flexibly adjusted according to the load balancing situation of the second type of computing device as the receiver, so as to balance the memory of each computing device as much as possible while reducing the memory pressure of each computing device.

[0112] The disclosed embodiments achieve load balancing of various computing devices based on the memory utilization of computing nodes and other conditions, and can be applied in large-scale deep learning model training, federated learning, and various fields involving distributed training. It also helps to promote the optimization of artificial intelligence acceleration cards and software, and promote the further development of artificial intelligence technology.

[0113] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0114] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0115] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0116] Refer to the following Fig.10 The electronic device 1000 according to this embodiment of the present invention will be described. Fig.10 The electronic device 1000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Fig.10 As shown, the electronic device 1000 is in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: the at least one processing unit 1010, the at least one storage unit 1020, and a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010).

[0118] The storage unit stores program codes, which can be executed by the processing unit 1010, so that the processing unit 1010 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1010 can perform the method shown in the embodiment of the present disclosure.

[0119] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202 , and may further include a read-only storage unit (ROM) 10203 .

[0120] The storage unit 1020 may also include a program / utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0121] Bus 1030 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0122] The electronic device 1000 may also communicate with one or more external devices 1100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1050. In addition, the electronic device 1000 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 through a bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0123] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0124] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0125] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0126] The program product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0127] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0128] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0129] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0130] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0131] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A model deployment method, characterized in that: include: Determine N data processing stages of a target model, wherein the target model is divided into the N data processing stages and runs in a pipeline parallel manner; Determining, according to the stage sequence number of the data processing stage, whether the data processing stage is a first type of data processing stage or a second type of data processing stage in the preheating stage, the first type of data processing stage is to activate the data transfer side, and the second type of data processing stage is to activate the data receiver; The first type of data processing stage and the second type of data processing stage are deployed in a computing node, and the computing node includes multiple computing devices, and the multiple computing devices include a first type of computing device where the first type of data processing stage is located and a second type of computing device where the second type of data processing stage is located.

2. The model deployment method according to claim 1, characterized in that: Determining, according to the stage sequence number of the data processing stage, whether the data processing stage is a first type of data processing stage or a second type of data processing stage in the preheating stage includes: When i<(N+1) / 2, the i-th data processing stage is the first type of data processing stage; When i ≥ (N+1) / 2, the i-th data processing stage is the second type of data processing stage; Among them, 1≤i≤N.

3. The model deployment method according to claim 1, characterized in that: The deploying the first type of data processing stage and the second type of data processing stage in one computing node comprises: Determine the memory usage value of the data processing stage in the preheating stage according to the stage sequence number of the data processing stage; Obtaining available memory of each of the computing devices; Determine the to-be-transferred memory value of the first type of computing device corresponding to the first type of data processing stage according to the difference between the memory occupancy value of the first type of data processing stage and the available memory of the computing device; Determining a receivable memory value of the second type of computing device corresponding to the second type of data processing stage according to a difference between the available memory of the computing device and the memory occupancy value of the second type of data processing stage; The multiple computing devices in a computing node are respectively set as the first category computing devices and the second category computing devices, the first category data processing stage is deployed on the first category computing devices, and the second category data processing stage is deployed on the second category computing devices, and the sum of the memory values ​​to be transferred corresponding to the first category computing devices in the computing node is less than the sum of the receivable memory values ​​corresponding to the second category computing devices.

4. The model deployment method according to claim 3, characterized in that: The deploying the first type of data processing stage and the second type of data processing stage in one computing node comprises: In the computing node, at least one computing device group for activating data transfer with a fixed matching relationship is determined according to the memory value to be transferred and the receivable memory value, and the computing device group includes the first type of computing devices and the second type of computing devices.

5. A data processing method, characterized in that: The target model is executed by the target model deployed by the model deployment method according to any one of claims 1 to 4, and the data processing method comprises: In response to a training data input message, receiving and processing training data; When it is determined that the warm-up phase has begun, the activation data of the first type of data processing phase is transferred from the first type of computing device to the second type of computing device for temporary storage in a computing node, the first type of computing device being the computing device where the first type of data processing phase is deployed, and the second type of computing device being the computing device where the second type of data processing phase is deployed; In a back-propagation phase, the activation data is retrieved from the second type of computing device to the first type of computing device.

6. The data processing method according to claim 5, characterized in that: The transmitting, within a computing node, the activation data of the first type of data processing stage from the first type of computing device to the second type of computing device for temporary storage comprises: When the first-type computing device forms activation data, the activation data is transmitted to the second-type computing device having a fixed matching relationship with the first-type computing device for temporary storage.

7. The data processing method according to claim 5, characterized in that: The transferring of the activation data of the first type of data processing stage from the first type of computing device to the second type of computing device for temporary storage in a computing node comprises: When it is determined that the warm-up phase has begun, determining a memory value to be transferred for each computing device of the first type in the computing node; Obtaining a receivable memory value of each of the second-category computing devices in the computing node; Determine the second-category computing device corresponding to each first-category computing device according to the to-be-transferred memory value and the receivable memory value; When the first type of computing device forms activation data, the activation data of the memory occupancy number corresponding to the memory value to be transferred is transmitted to the second type of computing device corresponding to the first type of computing device.

8. The data processing method according to claim 1, characterized in that: The transferring of the activation data of the first type of data processing stage from the first type of computing device to the second type of computing device for temporary storage in a computing node comprises: The activation data is transmitted via an inter-GPU NVLink link.

9. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 5 to 8 based on instructions stored in the memory.

10. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 5 to 8 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 5 to 8 are implemented.

Images