Data processing method, apparatus, and electronic device

Abstract

The application provides a data processing method and device, electronic equipment and computer storage medium. In the application, the neural network model is initially segmented based on available training resources and initial training parameters, the neural network model is pre-trained based on the initial segmentation result, and it is determined whether to re-segment the neural network model based on the pre-training result. In this way, the segmentation scheme of the neural network model is more matched with the available training resources and the neural network model, and the subsequent trained neural network model has good performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method, and the present application also relates to a device, an electronic device and a computer storage medium corresponding to the data processing method. BACKGROUND

[0002] A deep neural network model is a technique in the field of machine learning (ML). A neural network is a series of algorithms that recognize potential relationships in a set of data. Neural networks can adapt to changing inputs, generating optimal results without the need to redesign output criteria. In a way, these neural networks resemble biological neuronal systems. Today, deep neural network models are increasingly widely used. Before data is analyzed using a deep neural network model, the deep neural network model generally needs to be trained.

[0003] Therefore, in the field of deep neural network models, how to train a deep neural network model so that the deep neural network model can have good performance becomes a technical problem to be solved. SUMMARY

[0004] The present application provides a data processing method to solve the technical problem of how to train a deep neural network model so that the deep neural network model can have good performance. The present application also provides a data processing device, another data processing method, an electronic device and a computer storage medium.

[0005] The present application provides a data processing method, comprising:

[0006] determining available training resources and initial training parameters for training a neural network model;

[0007] performing initial segmentation on the neural network model according to the training resources and the initial training parameters, to obtain an initial segmentation result of the neural network model;

[0008] performing pre-training on the neural network model based on the initial segmentation result of the neural network model, to obtain a pre-training result;

[0009] determining whether the neural network model needs to be re-segmented according to the pre-training result, and if so, performing re-segmentation on the neural network model until a segmentation result that meets the training of the neural network model is obtained.

[0010] Optionally, the determination of the available training resources for training the neural network model comprises:

[0011] determining a network topology structure diagram of a training device used for training the neural network model;

[0012] performing analysis on the network topology graph by using a minimum cut recursive sorting algorithm to obtain a sorting of the plurality of training devices in the network topology graph;

[0013] determining available training resources of each of the training devices used for training the neural network model based on the sorting of the plurality of training devices.

[0014] Optionally, the initial training parameter comprises: segmentation information used for indicating that sample data used for training the neural network model is segmented.

[0015] the initial segmentation of the neural network model according to the training resources and the initial training parameter to obtain an initial segmentation result of the neural network model, comprises:

[0016] obtaining a segmented neural network model used for indicating that the neural network model is segmented and sequence number information of a training device corresponding to each model segment in the segmented neural network model based on the sorting of the plurality of training devices, the segmentation information used for indicating that sample data used for training the neural network model is segmented.

[0017] Optionally, the initial training parameter comprises: an initial segmentation number used for indicating that the neural network model is segmented.

[0018] the initial segmentation of the neural network model according to the training resources and the initial training parameter to obtain an initial segmentation result of the neural network model, comprises:

[0019] obtaining a segmented neural network model used for indicating that the neural network model is segmented and sequence number information of a training device corresponding to each model segment in the segmented neural network model based on the sorting of the plurality of training devices, the segmentation information used for indicating that sample data used for training the neural network model is segmented, and the initial segmentation number.

[0020] Optionally, the determining whether the neural network model needs to be re-segmented according to the pre-training result comprises:

[0021] determining whether a segmentation number corresponding to the segmented neural network model in the pre-training result is less than a number of layers of the neural network model.

[0022] Optionally, the re-segmenting the neural network model comprises:

[0023] add 1 to the number of the segmentation quantity corresponding to the segmented neural network model in the pre-training result as the initial segmentation quantity, to return to execute the step of obtaining the segmented neural network model for representing the neural network model being segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model based on the sorting of the plurality of training devices, the segmentation information for representing the sample data for training the neural network model being segmented, and the initial segmentation quantity.

[0024] Optionally, the pre-training of the neural network model based on the initial segmentation result of the neural network model comprises:

[0025] based on the segmented neural network model and the sequence number information of the training device corresponding to each model segment in the segmented neural network model, training the neural network model by the plurality of training devices for a pre-specified number of test rounds to obtain a current training duration;

[0026] determining whether the current training duration is less than a pre-set minimum training duration, and if so, updating the minimum training duration as the current training duration;

[0027] obtaining the segmented neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration;

[0028] based on the segmented neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration, training the neural network model by the plurality of training devices for a pre-specified number of rounds.

[0029] Optionally, it further comprises: obtaining the layer training duration of each layer of the neural network model on the training device.

[0030] the step of obtaining the segmented neural network model for representing the neural network model being segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model based on the sorting of the plurality of training devices, the segmentation information for representing the sample data for training the neural network model being segmented, and the initial segmentation quantity comprises:

[0031] based on the sorting of the plurality of training devices, the segmentation information for representing the sample data for training the neural network model being segmented, and the layer training duration, obtaining the segmented neural network model for representing the neural network model being segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model.

[0032] Optionally, the split information includes size information of the split sample data after the sample data is split.

[0033] The layer training duration of each layer of the neural network model on the training device is obtained, including:

[0034] Based on the size information of the split sample data, the neural network model is trained in a model parallel manner for a specified number of times, and the duration of forward propagation and back propagation of each layer of the neural network model is obtained. The duration of forward propagation and back propagation of each layer of the neural network model is used as the layer training duration.

[0035] Optionally, the split information for indicating that the sample data for training the neural network model is split includes: size information of the split sample data after the sample data is split and sequence number information of the split sample data.

[0036] Based on the ordering of the plurality of training devices, the split information for indicating that the sample data for training the neural network model is split, the split neural network model for indicating that the neural network model is split and the sequence number information of the training device corresponding to each model segment in the split neural network model are obtained, including:

[0037] Based on the size information of the split sample data and the sequence number information of the split sample data, the split neural network model for indicating that the neural network model is split is obtained.

[0038] Based on the ordering of the plurality of training devices and the split neural network model for indicating that the neural network model is split, the sequence number information of the training device corresponding to each model segment in the split neural network model is obtained.

[0039] Optionally, the network topology structure diagram of the training device for training the neural network model includes:

[0040] The training device in the system is taken as a node, and the communication bandwidth between the training devices is taken as the weight of the edge between the nodes. The network topology structure diagram of the training device for training the neural network model is determined.

[0041] The application further provides a data processing method, including:

[0042] Obtaining initial training parameters and training accuracy requirements of the user for training the neural network model through the client;

[0043] Determining available training resources for training the neural network model;

[0044] Performing initial segmentation on the neural network model according to the training resources and the initial training parameters to obtain an initial segmentation result of the neural network model;

[0045] Based on the initial segmentation result of the neural network model, pre-training the neural network model to obtain a pre-training result;

[0046] Determining whether the neural network model needs to be re-segmented based on the pre-training result, and if so, re-segmenting the neural network model until a segmentation result that satisfies the training of the neural network model is obtained;

[0047] Segmenting the neural network model based on the segmentation results, and deploying the segmented segments to distributed computing nodes for training to obtain a target neural network model that meets user requirements;

[0048] Return the target neural network model to the client.

[0049] The present application also provides an electronic device, comprising:

[0050] processor;

[0051] The memory is used to store a computer program, which is executed by the processor to perform the data processing method.

[0052] The present application also provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is run by a processor to execute the data processing method.

[0053] Compared with the prior art, the embodiments of the present application have the following advantages:

[0054] The present application provides a data processing method, including: determining the available training resources and initial training parameters for training a neural network model; performing an initial segmentation on the neural network model based on the training resources and the initial training parameters to obtain an initial segmentation result of the neural network model; pre-training the neural network model based on the initial segmentation result of the neural network model to obtain a pre-training result; determining whether the neural network model needs to be re-segmented based on the pre-training result, and if so, re-segmenting the neural network model until a segmentation result that satisfies the training of the neural network model is obtained. In the present application, the neural network model is firstly initially segmented based on the available training resources and the initial training parameters, the neural network model is pre-trained based on the initial segmentation result, and whether the neural network model needs to be re-segmented is determined based on the pre-training result. In fact, this method continuously optimizes the segmentation scheme for segmenting the neural network model, which can make the segmentation scheme of the neural network model more compatible with the available training resources and the neural network model, so that the subsequently trained neural network model has good performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0056] Figure 1A A schematic diagram of different training methods for deep learning models;

[0057] Figure 1B The first schematic diagram of the pipelined parallel training method for deep learning models;

[0058] Figure 1C A second schematic diagram of a pipelined parallel training method for a deep learning model;

[0059] Figure 1 A flowchart of a data processing method provided in the first embodiment of the present application;

[0060] Figure 2 A schematic diagram of a data processing device provided in accordance with a second embodiment of the present application;

[0061] Figure 3 A schematic diagram of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0062] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0063] The present application provides a data processing method, device, electronic equipment and computer storage medium. The following will introduce the data processing method, device, electronic equipment and computer storage medium through specific embodiments.

[0064] In order to facilitate the understanding of the data processing method provided by the present application, the following will preliminarily explain the terms involved in the data processing method of the present application.

[0065] The data processing method of the present application is mainly used in the scene of training a neural network model. The data processing method of the present application is mainly used to introduce how to segment the model, and then allocate the segmented model fragments to each training device for training. Of course, in the actual training process, the training sample fragments can also be allocated to each training device for training.

[0066] Deep learning distributed training: a training method using multiple computing devices (computing devices are training devices) or computing nodes to accelerate parallel computing of deep learning models (deep learning models are deep neural network models).

[0067] Model parallel: a parallel way of deep learning distributed training, which is mainly used to split the model structure to multiple computing nodes or computing devices.

[0068] Data parallel: a parallel way of deep learning distributed training, which is mainly used to copy the entire model to multiple computing nodes or computing devices, but each computing node or computing device reads different data.

[0069] Deep learning operator: refers to the basic computing granularity of deep learning model, some are mathematical operations or logical operations (such as addition, subtraction, multiplication, division operations, and and, or, not and exclusive or operations), and some are other basic operations of vectors or matrices (such as matrix multiplication).

[0070] Tensor: that is, Tensor, is the basic data structure used in deep learning model calculation, which is a multi-dimensional vector. This is mainly for data, which generally needs to be converted into a multi-dimensional vector for processing. For example, an image with a resolution of 256*256 can be converted into a 256*256*3-dimensional vector for processing, where 256*256 is the number of pixels, and 3 is the color of each pixel, which consists of red, green and blue.

[0071] Tensor dimension: It is a mathematical concept, specifically refers to a certain dimension of a tensor.

[0072] Tensor split: It is an operation that splits a tensor into multiple tensors according to one or more dimensions.

[0073] Operator split: It is a special form of model parallel, which can refer to the splitting of deep learning operators caused by tensor splitting. Usually after splitting, a single operator becomes multiple operator representations, and this splitting process must ensure the equivalence of mathematical calculations before and after splitting. After splitting, multiple operators and tensors will be placed on different computing devices or computing nodes.

[0074] Pipeline parallel: A special form of model parallel, which can refer to splitting each part of the deep learning model to different computing devices or computing nodes, so that a single set of data is calculated in sequence on multiple computing devices or computing nodes, and different sets of data are parallel. Parallel way. A single set of data is obtained by dividing a batch into multiple micro-batches.

[0075] The main differences between model parallel and pipeline parallel are as follows: First, model parallel directly inputs a sample batch for training; while pipeline parallel divides the sample batch into multiple micro-batches, and then inputs them into the system containing multiple training devices for training. Second, in pipeline parallel, a model segment can be trained on multiple devices, while in model parallel, each model segment is trained on only one device.

[0076] Figure 1A A schematic diagram of different training methods for deep learning models. It shows a model being trained on three computing devices (GPU1, GPU2, and GPU3) in different parallel ways, where a batch is divided into three micro-batches.

[0077] Distributed strategy automatic exploration: It refers to the process of automatically exploring the distributed training method suitable for the hardware topology and the current model characteristics. The final explored training method can include data parallel, model parallel, pipeline parallel, or a combination of two or more.

[0078] Distributed machine learning aims to utilize multiple computing devices to train large deep learning neural networks. Common distributed machine learning methods include data parallelism, model parallelism, and pipeline parallelism. Data parallelism divides the entire dataset (which can refer to a sample set) to multiple training devices (e.g., GPUs). Each training device trains in its own training set and synchronizes the neural network parameters after each training device trains a sample batch. Model parallelism divides the entire deep neural network model to each training device, and each model segment on each training device contains multiple layers of the original model in sequence. The training of each training sample batch is calculated in sequence by each model segment on each training device, with forward propagation followed by backward propagation.

[0079] Pipeline parallelism combines data parallelism and model parallelism by dividing the model to multiple computing devices. Unlike model parallelism, each model segment can be replicated to multiple devices and processed in a data-parallel manner during training. Meanwhile, pipeline parallelism further divides each sample batch into micro-batches of equal size, and then sequentially adds each micro-batch to the training process, so that different training devices can process different micro-batches simultaneously, improving system computing resource utilization. In order to ensure the convergence and accuracy of model training, the synchronization wall is inserted into the pipeline when all micro-batches belonging to the same batch are trained, to ensure that the results of the entire pipeline parallel training are equivalent to the results of training directly based on the batch.

[0080] Figure 1B With Figure 1C respectively, the first and second schematic diagrams of the pipeline parallel training method of the deep learning model. A deep learning model is divided into three model segments and placed on four GPUs (computing devices). Among them, the second segment is placed on GPU 2 and GPU 3 at the same time. The scheduling manner is as follows Figure 1C , where the second model segment is calculated in a data-parallel manner on GPU 2 and GPU 3 at the same time, and finally the parameters need to be synchronized through the AllReduce operator.

[0081] The data processing method of the present application can explore a suitable way to train the neural network model based on the network topology of the training device and the characteristics of the neural network model.

[0082] First embodiment

[0083] The first embodiment of the present application provides a data processing method, which will be described below in combination with Figure 1 .

[0084] Please refer to Figure 1 , which is a flowchart of the data processing method provided by the first embodiment of the present application.

[0085] The data processing method of the embodiment of the application comprises the following steps:

[0086] Step S101: determining available training resources and initial training parameters for training the neural network model.

[0087] The available training resources for training the neural network model can be the available training resources of each training device in the training device used for training the neural network model. As one way of determining the available training resources of each training device in the training device used for training the neural network model, first, the network topology graph of the training device used for training the neural network model is determined; then, the network topology graph is analyzed by using a minimum cut recursive sorting algorithm to obtain the sorting of the plurality of training devices in the network topology graph; finally, based on the sorting of the plurality of training devices, the available training resources of each training device in the training device used for training the neural network model are determined.

[0088] In the embodiment, as one implementation of determining the network topology graph of the training device used for training the neural network model, the training devices in the system are taken as nodes, and the communication bandwidth between the training devices is taken as the weight of the edge between the nodes, to determine the network topology graph of the training device used for training the neural network model.

[0089] In the embodiment, as one implementation of analyzing the network topology graph by using the minimum cut recursive sorting algorithm to obtain the sorting of the plurality of training devices in the network topology graph, it is judged whether there is only one node in the network topology graph, and if not, the network topology graph is analyzed by using the minimum cut recursive sorting algorithm to obtain the sorting of the plurality of training devices in the network topology graph.

[0090] The sorting of the plurality of training devices obtained in this step can make full use of the inter-device connection (inter-GPU NVLink) with high bandwidth, and reduce the communication time in the subsequent design of the training scheme.

[0091] The available training resources include at least one of the following resources: the processor of the training device, the internal storage space size of the training device, and the communication mode between the training devices. For example, the CPU and the memory of the computing device, and the connection mode used for communication between the computing devices.

[0092] In the embodiment, the initial training parameters can be some training parameters specified by the user on the client side for pre-training the neural network model. The pre-training is a training process before the formal training of the neural network model. The purpose of the pre-training process of the application is to obtain suitable training parameters and suitable segmentation results for segmenting the neural network model.

[0093] The initial training parameter can include segmentation information indicating that the sample data used for training the neural network model is segmented, an initial segmentation number indicating that the neural network model is segmented, and the like.

[0094] Step S102: performing initial segmentation on the neural network model according to the training resource and the initial training parameter to obtain an initial segmentation result of the neural network model.

[0095] As a manner of performing initial segmentation on the neural network model according to the training resource and the initial training parameter to obtain an initial segmentation result of the neural network model, it can refer to:

[0096] Based on the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented, a segmented neural network model indicating that the neural network model is segmented and sequence number information of the training device corresponding to each model segment in the segmented neural network model are obtained.

[0097] Specifically, as an embodiment of obtaining the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model based on the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented, a recursive ranking algorithm is used to analyze the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented to obtain the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model.

[0098] In this embodiment, as an embodiment of obtaining the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model based on the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented, the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model are obtained based on the ranking of the plurality of training devices, the segmentation information indicating that the sample data used for training the neural network model is segmented, and an initial segmentation number indicating that the neural network model is segmented. For example, the initial segmentation number can be 1.

[0099] In the embodiment, the sequence number information of the training device corresponding to each model segment of the segmented neural network model can also be obtained based on the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented, in the following manner. Specifically, first, the layer training duration of each layer of the neural network model on the training device is obtained. Then, based on the ranking of the plurality of training devices, the segmentation information indicating that the sample data used for training the neural network model is segmented, and the layer training duration, the sequence number information of the training device corresponding to each model segment of the segmented neural network model is obtained.

[0100] In the embodiment, the segmentation information includes size information of the segmented sample data after the sample data is segmented. Therefore, as the layer training duration of each layer of the neural network model on the training device, the time duration of the forward propagation and the backward propagation of each layer of the neural network model can be obtained by training the neural network model a specified number of times in a model parallel manner based on the size information of the segmented sample data, and the time duration of the forward propagation and the backward propagation of each layer of the neural network model is taken as the layer training duration.

[0101] In addition, in the embodiment, the segmentation information indicating that the sample data used for training the neural network model is segmented includes size information of the segmented sample data after the sample data is segmented and sequence number information of the segmented sample data. Therefore, as the sequence number information of the training device corresponding to each model segment of the segmented neural network model obtained based on the ranking of the plurality of training devices and the segmentation information indicating that the sample data used for training the neural network model is segmented, the sequence number information of the training device corresponding to each model segment of the segmented neural network model can be obtained in the following manner. First, based on the size information of the segmented sample data and the sequence number information of the segmented sample data, the segmented neural network model indicating that the neural network model is segmented is obtained. Then, based on the ranking of the plurality of training devices and the segmented neural network model indicating that the neural network model is segmented, the sequence number information of the training device corresponding to each model segment of the segmented neural network model is obtained.

[0102] Step S103: pre-training the neural network model based on the initial segmentation result of the neural network model to obtain a pre-training result.

[0103] As the pre-training of the neural network model based on the initial segmentation result of the neural network model, the following can be referred to.

[0104] First, based on the segmented neural network model and the serial number information of the training device corresponding to each model segment in the segmented neural network model, a plurality of training devices are used to train the neural network model for a pre-specified number of test rounds to obtain a current training duration.

[0105] Then, it is judged whether the current training duration is less than the pre-set minimum training duration. If yes, the minimum training duration is updated to the current training duration.

[0106] After that, the segmented neural network model corresponding to the current training duration and the serial number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration are obtained.

[0107] Finally, based on the segmented neural network model corresponding to the current training duration and the serial number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration, a plurality of training devices are used to train the neural network model for a pre-specified number of rounds.

[0108] Based on the initial segmentation result of the neural network model, the neural network model is pre-trained, comprising:

[0109] Based on the segmented neural network model and the serial number information of the training device corresponding to each model segment in the segmented neural network model, a plurality of training devices are used to train the neural network model for a pre-specified number of test rounds to obtain a current training duration;

[0110] It is judged whether the current training duration is less than the pre-set minimum training duration. If yes, the minimum training duration is updated to the current training duration.

[0111] The segmented neural network model corresponding to the current training duration and the serial number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration are obtained.

[0112] Based on the segmented neural network model corresponding to the current training duration and the serial number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration, a plurality of training devices are used to train the neural network model for a pre-specified number of rounds.

[0113] Step S104: According to the pre-training result, it is determined whether the neural network model needs to be re-segmented. If yes, the neural network model is re-segmented until a segmentation result that meets the training of the neural network model is obtained.

[0114] Specifically, the determining whether the neural network model needs to be re-partitioned according to the pre-training result can refer to: determining whether the number of partitions corresponding to the partitioned neural network model in the pre-training result is less than the number of layers of the neural network model.

[0115] If the neural network model needs to be re-partitioned, as the re-partitioning of the neural network model, it can refer to: adding 1 to the number of partitions corresponding to the partitioned neural network model in the pre-training result as an initial partition number, returning to execute the steps of sorting based on multiple training devices, partition information for indicating that the sample data for training the neural network model is partitioned, the initial partition number, obtaining the partitioned neural network model for indicating that the neural network model is partitioned and the sequence number information of the training device corresponding to each model segment in the partitioned neural network model.

[0116] In this embodiment, the above method is actually based on dynamic planning of deep neural network model segmentation and device deployment, and the dynamic planning of deep neural network model segmentation and device deployment can be implemented in the following way.

[0117] First, the number of partitions for indicating that the neural network model is partitioned is obtained.

[0118] Then, it is determined whether the number of partitions is less than the number of layers of the neural network model. If yes, 1 is added to the number of partitions, and the step of obtaining the partitioned neural network model for indicating that the neural network model is partitioned and the sequence number information of the training device corresponding to each model segment in the partitioned neural network model is returned.

[0119] As an embodiment of training the neural network model by using multiple training devices based on the partitioned neural network model and the sequence number information of the training device corresponding to each model segment in the partitioned neural network model, it can refer to:

[0120] First, based on the partitioned neural network model and the sequence number information of the training device corresponding to each model segment in the partitioned neural network model, the neural network model is trained by using multiple training devices for a pre-specified number of test rounds to obtain a current training duration.

[0121] Then, it is determined whether the current training duration is less than the pre-set minimum training duration. If yes, the minimum training duration is updated to the current training duration.

[0122] Further, the partitioned neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the partitioned neural network model corresponding to the current training duration are obtained.

[0123] Finally, based on the segmented neural network model corresponding to the current training duration and the serial number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration, the multiple training devices are used to train the neural network model for a pre-specified number of rounds.

[0124] In fact, in the present embodiment, it is necessary to sample the training time of the deep neural network, specifically: the user gives the micro-batch size micro_batch_size, the micro-batch number num_micro_batch of single batch division, and an L-layer deep neural network. That is, the calculation time (i.e., layer training duration) exec_time of each layer of the deep neural network on the calculation device under the condition of user-given micro_batch_size can be obtained based on the sampling method. The exec_time[l] of each layer of the network on the calculation device is obtained for model segmentation and device deployment.

[0125] In obtaining the segmented neural network model for representing the segmentation of the neural network model and the serial number information of the training device corresponding to each model segment in the segmented neural network model, multiple segmentation schemes will be traversed. For each segmentation scheme, it is necessary to actually train the model to decide how the actual training time of a certain scheme is. The num_trial_iteration (pre-specified test number of rounds) is the number of test training rounds to judge how the current scheme training time is. Numerically, num_trial_iteration should be much smaller than num_training_iteration. Num_training_iteration is the total training number of rounds required for actual training, that is, the total number of training rounds (Num_training_iteration is the pre-specified number of rounds).

[0126] The micro-batch size is the size of the segmented sample data for representing the segmented sample data as described above, and the micro-batch number is the serial number of the segmented sample data.

[0127] The deep neural network training time sampling step specifically comprises: first, training the deep neural network profile_time times in a model-parallel manner with a micro batch size micro_batch_size, wherein profile_time is a user-specified parameter. In practice, the running time of each layer on the current training device needs to be recorded by actually running the model several times. The number of rounds of running is determined by profile_time, because this profile is only one-time, so profile_time can be larger than num_trial_iteration, thereby ensuring the accuracy of the statistical running time as the basis for subsequent network splitting and device deployment. Network splitting and device deployment are to obtain a split neural network model for representing a neural network model that is split, and sequence number information of a training device corresponding to each model segment in the split neural network model.

[0128] Then, the total time of forward propagation and backward propagation of each layer of the deep neural network in profile_time training is statistically counted, and the data volume transmitted between each two consecutive layers of the model is also counted. exec_time[l] records the training time of the lth layer, and data[l, l+1] records the data communication volume between layer l and layer l+1.

[0129] The network topology graph is analyzed by using a minimum cut recursive sorting algorithm to obtain the sorting of the plurality of training devices in the network topology graph. In fact, this is a step of calculating device sorting, and specifically, the device sorting step is designed based on a recursive method, and let recursive_device_ordering(device_graph, low_rank, high_rank) represent the function corresponding to the device sorting step. Wherein, device_graph is a directed acyclic graph abstracted based on all computing devices in the current system, the nodes in the graph are training devices, and there is an edge between each two nodes, and the weight of the edge is the minimum communication bandwidth between the nodes. For example, the edge weight between GPUs of the same type of server is the intra-machine bandwidth PCIE or NVLink, and the edge weight between GPUs of different types of servers is the network bandwidth. The core of the minimum cut algorithm is: input a graph, divide this graph into two independent graphs, and minimize the sum of the weights of the edges cut in the division (these edges connect the two graphs after division). By using this feature, the communication bandwidth between nodes is used as the weight of the edge, and it is ensured that the edges cut each time are edges with relatively small bandwidth.

[0130] This step is mainly to initialize num_stage_all, i.e., the total number of model splits, which should be between 1 and the number of model layers. By traversing from 1 to the number of model layers, the best split number is determined.

[0131] In particular, the above configuration sets the device graph between the current system training devices, sets low_rank as 1, and high_rank as the total number of training devices in the current system. Then, recursive_device_ordering(device_graph, low_rank, high_rank) is executed. recursive_device_ordering(device_graph, low_rank, high_rank).

[0132] recursive_device_ordering(device_graph, low_rank, high_rank) is executed. The specific process of recursive_device_ordering(device_graph, low_rank, high_rank) includes:

[0133] If there is only one training device node in device_graph, set device_order[low_rank] as the id of the training device.

[0134] Otherwise, perform a minimum cut on the current device_graph to obtain two graphs graph_1 and graph_2. Set low_rank_1 as low_rank, high_rank_1 as low_rank_1 plus the number of nodes in graph_1 minus 1; low_rank_2 as low_rank_1 plus the number of nodes in graph_1, and high_rank_2 as high_rank; then, execute recursive_device_ordering(graph_1, low_rank_1, high_rank_1); and execute recursive_device_ordering(graph_2, low_rank_2, high_rank_2).

[0135] The above step S103 is actually the step of deep neural network model segmentation and device deployment. This step is actually designed based on a recursive method. Let pipeline_partition_replication_mapping(device_graph, device_order, num_remain_layer, num_remain_stage, num_remain_GPU, num_last_stage_replica) represent the function corresponding to the deep neural network model segmentation and device deployment steps. It means that the first 1 to num_remain_layer layers in the deep neural network are divided into num_remain_stage segments and placed on num_remain_GPU GPUs, and the last model segment is trained in a data-parallel manner on num_last_stage_replica GPUs. The specific process is as follows:

[0136] First, the user defines the minimum min_time, which is the maximum of the initial minimum training time for a single model slice across all micro-batches or the total micro-batch communication time between two consecutive model slices. The initial minimum training time for a single model slice across all micro-batches is essentially the training time for all micro-batches for any single model slice, while the total micro-batch communication time between two consecutive model slices is the total communication time between any two model slices.

[0137] by Figure 1C For example, this is the maximum of the five times: the total forward and backward execution time of three micro-batches on model fragment 1 (excluding any idle time in between), the total execution time of three micro-batches on model fragment 2 (i.e., the time it takes for either GPU2 or GPU3 (since they are both executing simultaneously) to execute all three micro-batches, plus the time for the AllReduce operator), the total computation time of model fragment 3, the total communication time between model fragments 1 and 2, and the total communication time between model fragments 2 and 3. Because we're looking for a minimum value here, we can initialize min_time to INF, where INF is a user-defined large value that exceeds the number of possible values ​​in system calculations.

[0138] Initialize num_remain_layer to the total number of layers in the current network, initialize num_remain_stage to the current num_stage_all, initialize num_remain_GPU to the total number of training devices in the current system, and initialize num_last_stage_replica to 1. The above communication time mainly refers to the transmission time required for two consecutive network slices to transmit data during the training process. For example, Figure 1CThe time of the part of the transmission corresponding to the middle arrow is the communication time.

[0139] Then, pipeline_partition_replication_mapping(device_graph, device_order, num_remain_layer, num_remain_stage, num_remain_GPU, num_last_stage_replica) is executed, and the return value is max_single_stage_time, which is the larger one of the maximum time of training all micro-batches of all single model segments and the maximum communication time between any two consecutive model segments, and device_placement and stage_set. The stage_set stores the result of the neural network partitioning. The neural network is divided into num_stage_all model segments, and stage_set[i] represents the network layer number contained in the i-th model segment. For example, a 6-layer neural network is divided into 3 model segments, and stage_set[1] = [1, 2], stage_set[2] = [3, 4], and stage_set[3] = [5, 6].

[0140] If max_single_stage_time is less than min_time, min_time is set to max_single_stage_time, and the current device_placement and stage_set are recorded.

[0141] If num_last_stage_replica is less than the total number of training devices in the current system, num_last_stage_replica is increased by 1, and pipeline_partition_replication_mapping(device_graph, device_order, num_remain_layer, num_remain_stage, num_remain_GPU, num_last_stage_replica) is executed again. The return value is max_single_stage_time, which is the larger one of the maximum time of training all micro-batches of all single model segments and the maximum communication time between any two consecutive model segments, and device_placement and stage_set. Otherwise, the device_placement and stage_set that achieve the minimum max_single_stage_time are returned.

[0142] The above pipeline_partition_replication_mapping(device_graph, device_order, num_remain_layer, num_remain_stage, num_remain_GPU, num_last_stage_replica) returns a value of max_single_stage_time, which is the larger one of the maximum value of the time of training all micro-batches of all single model segments and the maximum value of the communication time of all micro-batches between any two consecutive model segments, and device_placement and stage_set. The pipeline_partition_replication_mapping(device_graph, device_order, num_remain_layer, num_remain_stage, num_remain_GPU, num_last_stage_replica) specifically includes the following steps.

[0143] First, if num_remain_layer or num_remain_GPU is less than num_remain_stage, max_single_stage_time is set to INF, device_placement and stage_set are empty, and the return is made. Otherwise, if num_remain_stage is 1 and num_last_stage_replica is equal to num_remain_GPU, max_single_stage_time is calculated in the following manner:

[0144]

[0145] where M is num_micro_batch, p i is exec_time[i], n is num_remain_GPU, and l is num_remain_layer. AllReduce 1,l (1, n) is the time of deploying the depth neural network layers 1 to l as a stage on the GPUs of device_order[1] to device_order[n] using AllReduce communication, which is calculated in the following manner:

[0146]

[0147] where l1, l2 are the start and end layers of the model segment of the deep neural network respectively, n1, n2 are the training devices from device_order[n1] to device_order[n2] that the model segment is deployed on. k is n2-n1+1, a i is the size of the i-th layer of the deep neural network parameters, b min is the minimum communication bandwidth between all training devices from device_order[n1] to device_order[n2]. After max_single_stage_time is calculated, stage_set[1] is set as [1, 2, …, num_remain_layer] and device_placement[1] is set as [device_order[1], …, device_order[n]]. Return; otherwise, if num_remain_stage is 1 or num_last_stage_replica is equal to num_remain_GPU, set max_single_stage_time as INF, device_placement and stage_set as empty, and return; otherwise, set min_max_time as INF.

[0148] Then, initialize current_num_remain_layer as 1 and current_num_last_stage_replica as 1.

[0149] Further, process the sub-problem, call pipeline_partition_replication_mapping(device_graph, device_order, current_num_remain_layer, num_remain_stage-1, num_remain_GPU-num_last_stage_replica, current_num_last_stage_replica), and obtain the return values current_max_single_stage_time, device_placement and stage_set. Calculate the communication time comm_time between the last model segment in the sub-problem and the currently processed model segment. Specifically, comm_time is obtained by the following formula:

[0150]

[0151] This part adopts dynamic programming algorithm to get the optimal solution by repeatedly recursive calling pipeline_partition_replication_mapping(). Here the sub-problem is to calculate the optimal deployment of cutting current_num_remain_layer into num_remain_stage-1 model fragments and placing them on num_remain_GPU-num_last_stage_replica GPUs, and the last model fragment is deployed on current_num_last_stage_replica GPUs by recursive calling pipeline_partition_replication_mapping().

[0152] where d l′,l′+1 is data[current_num_remain_layer,current_num_remain_layer+1], r stage_min is the minimum value of the bandwidth between each two of the training devices from device_order[num_remain_GPU-num_last_stage_replica-current_num_remain_layer+1] to device_order[num_remain_GPU-num_last_stage_replica] and device_order[num_remain_GPU-num_last_stage_replica+1] to device_order[num_remain_GPU]. At the same time, the calculation time comp_time of current current_num_remain_layer+1 layer to num_remain_layer layer as a model fragment placed on device_order[num_remain_GPU-num_last_stage_replica+1] to device_order[num_remain_GPU] training devices is calculated. Specifically, comp_time is obtained by the following calculation formula:

[0153]

[0154] wherein l' is current_num_remain_layer. Let temp_max be the maximum of current_max_single_stage_time, comm_time and comp_time.

[0155] After that, if min_max_time is greater than temp_max, set min_max_time to temp_max, stage_set[num_remain_stage] to [current_num_remain_layer+1,..., num_remain_layer], and device_placement[num_remain_stage] to [device_order[num_remain_GPU-num_last_stage_replica+1],..., device_order[num_remain_GPU]].

[0156] Further after that, if current_num_last_stage_replica is less than num_remain_GPU-num_last_stage_replica, then current_num_last_stage_replica is incremented by 1, and the process of if min_max_time is greater than temp_max, set min_max_time to temp_max, stage_set[num_remain_stage] to [current_num_remain_layer+1,..., num_remain_layer], and device_placement[num_remain_stage] to [device_order[num_remain_GPU-num_last_stage_replica+1],..., device_order[num_remain_GPU]] is continued. Otherwise, current_num_last_stage_replica is reset to 1, and if current_num_remain_layer is less than num_remain_layer-1, then current_num_remain_layer is incremented by 1; otherwise, the current min_max_time, stage_set and device_placement are returned.

[0157] In one training round, the order of training each micro-batch forward and backward propagation on each model segment is determined: the queue stage_execution_order[s] represents the ordered list of the execution order of the s-th model segment, for example: if num_micro_batch = 2, one order is stage_execution_order[s] = [[1, F], [2, F], [1, B], [2, B]], where [m, F] ([m, B]) represents the execution of the forward propagation (backward propagation) training of the m-th micro-batch.

[0158] Then, the training is performed according to the stage_execution_order calculated by the above process, and the training is performed for a given number of rounds.

[0159] The specific steps of the above process for determining the order of training each micro-batch forward and backward propagation on each model segment in a single training round are as follows: where num_stage represents the number of network segments into which the current deep neural network is divided.

[0160] First, initialize stage_available_micro_batch, where each element represents all network segments that can be processed but have not been processed by a network segment. Among them, stage_available_micro_batch[1] is initialized as [[1, F], …, [num_micro_batch, F]], that is, all micro-batches can be executed in the forward calculation of the first network segment at the beginning. The rest of stage_available_micro_batch[2] to stage_available_micro_batch[num_stage] is initialized as empty.

[0161] Then, initialize the auxiliary array interstage_comm_available_micro_batch, each element of which is empty. Each element interstage_comm_available_micro_batch[s, s+1] is a queue representing the current executable communication of the s-th network and the s+1-th network segment. Similarly, [m, F] ([m, B]) represents the data transmission in the execution of the m-th micro-batch forward propagation (backward propagation). The auxiliary array is an intermediate temporary variable (which will not appear in the final output of the function), and each element is a queue (satisfying the characteristic that the data is first in first out). One element interstage_comm_available_micro_batch[s, s+1] represents the current executable communication number of the s-th network and the s+1-th network segment.

[0162] After that, initialize stage_execution_order as empty.

[0163] After that, traverse stage_available_micro_batch[s] in order of s from 1 to num_stage. For the current stage_available_micro_batch[s], select one element [m, F] (if any) and remove it from stage_available_micro_batch[s] and add it to stage_execution_order[s]; then select one element [m, B] (if any) and remove it from stage_available_micro_batch[s] and add it to stage_execution_order[s].

[0164] After that, traverse interstage_comm_available_micro_batch[s, s+1] in order of s from 1 to num_stage-1. For the current interstage_comm_available_micro_batch[s, s+1], select one element [m, F] (if any) and remove it from interstage_comm_available_micro_batch[s, s+1]; then select one element [m, B] (if any) and remove it from interstage_comm_available_micro_batch[s, s+1].

[0165] After that, traverse in order of s from 1 to num_stage. If there is an element [m, F] removed from stage_available_micro_batch[s] and s is not equal to num_stage, then add [m, F] to interstage_comm_available_micro_batch[s, s+1]; if there is an element [m, B] removed from stage_available_micro_batch[s] and s is not equal to 1, then add [m, B] to interstage_comm_available_micro_batch[s-1, s].

[0166] Afterwards, traverse in order of s from 1 to num_stage-1, if there is an element [m, F] removed from interstage_comm_available_micro_batch[s, s+1], add [m, F] to stage_available_micro_batch[s+1], if s is num_stage-1, then add [m, B] to stage_available_micro_batch[s+1] again; if there is an element [m, B] removed from interstage_comm_available_micro_batch[s, s+1], add [m, B] to stage_available_micro_batch[s-1].

[0167] If there is still micro batch m whose execution order [m, B] on any network segment s has not been added to stage_available_micro_batch[s], continue to traverse stage_available_micro_batch[s] in order of s from 1 to num_stage. For the current stage_available_micro_batch[s], select an element [m, F] (if any) and remove it from stage_available_micro_batch[s] and add it to stage_execution_order[s]; then select an element [m, B] (if any) and remove it from stage_available_micro_batch[s] and add it to stage_execution_order[s]. Otherwise, for each network segment s to which multiple training devices are assigned, add [AllReduce] to stage_available_micro_batch[s] to indicate that the data parallel training of the segment is finally synchronized with parameters using AllReduce.

[0168] Since in the present application, the neural network model is initially segmented based on the available training resources and the initial training parameters, the neural network model is pre-trained based on the initial segmentation result, and whether the neural network model needs to be re-segmented is determined based on the pre-training result, in this way, the segmentation scheme of the neural network model can be more matched with the available training resources and the neural network model, so that the subsequent trained neural network model has good performance. Further, in the present application, the network topology graph of the training device used for training the neural network model is analyzed by using the minimum cut recursive sorting algorithm to obtain the sorting of the plurality of training devices in the network topology graph. Based on the minimum cut recursive sorting algorithm, the network bandwidth in the system composed of the training devices can be fully utilized, and the communication overhead in the training is reduced. At the same time, this method does not require that the network environment of the training device must be homogeneous, and the requirement for the network environment for training is not strict.

[0169] Second embodiment

[0170] Corresponding to the data processing provided by the first embodiment of the present application, the second embodiment of the present application also provides a data processing device. Since the device embodiment is basically similar to the first embodiment, it is described simply, and the related parts are referred to the part of the first embodiment. The device embodiment described below is only illustrative.

[0171] Please refer to Figure 2 which is a schematic diagram of the data processing device provided by the second embodiment of the present application.

[0172] The data processing device comprises:

[0173] The resource and parameter determination unit 201 is configured to determine the available training resources and the initial training parameters for training the neural network model.

[0174] The initial segmentation unit 202 is configured to perform initial segmentation on the neural network model according to the training resources and the initial training parameters, and obtain an initial segmentation result of the neural network model.

[0175] The pre-training result obtaining unit 203 is configured to pre-train the neural network model based on the initial segmentation result of the neural network model, and obtain a pre-training result.

[0176] The re-segmentation unit 204 is configured to determine whether the neural network model needs to be re-segmented according to the pre-training result, and if so, to re-segment the neural network model until a segmentation result that meets the training of the neural network model is obtained.

[0177] Optionally, the resource and parameter determination unit is specifically configured to:

[0178] determine a network topology graph of training devices for training the neural network model;

[0179] analyze the network topology graph by using a min-cut recursive sorting algorithm to obtain a sorting of the plurality of training devices in the network topology graph;

[0180] based on the sorting of the plurality of training devices, determine available training resources of each of the training devices for training the neural network model.

[0181] Optionally, the initial training parameters include: segmentation information indicating that sample data for training the neural network model is segmented.

[0182] The initial segmentation unit is specifically configured to:

[0183] based on the sorting of the plurality of training devices, the segmentation information indicating that sample data for training the neural network model is segmented, and the initial segmentation quantity, obtain segmentation information indicating that the neural network model is segmented and sequence number information of training devices corresponding to each model segment in the segmented neural network model.

[0184] Optionally, the initial training parameters include: an initial segmentation quantity indicating that the neural network model is segmented.

[0185] The initial segmentation unit is specifically configured to:

[0186] based on the sorting of the plurality of training devices, the segmentation information indicating that sample data for training the neural network model is segmented, and the initial segmentation quantity, obtain segmentation information indicating that the neural network model is segmented and sequence number information of training devices corresponding to each model segment in the segmented neural network model.

[0187] Optionally, the re-segmentation unit is specifically configured to:

[0188] determine whether a segmentation quantity corresponding to the segmented neural network model in the pre-training result is less than a layer number of the neural network model.

[0189] Optionally, the re-segmentation unit is specifically configured to:

[0190] Add 1 to the number of the split quantity corresponding to the split neural network model in the pre-training result as the initial split quantity, to return to execute the step of obtaining the split information for indicating that the sample data for training the neural network model is split, the initial split quantity, the split neural network model for indicating that the neural network model is split, and the sequence number information of the training device corresponding to each model segment in the split neural network model based on the ordering of the plurality of training devices.

[0191] Optionally, the pre-training result obtaining unit is specifically used for:

[0192] Based on the split neural network model and the sequence number information of the training device corresponding to each model segment in the split neural network model, a pre-specified number of training rounds of the neural network model is trained by using the plurality of training devices to obtain the current training duration.

[0193] If the current training duration is less than the pre-set minimum training duration, the minimum training duration is updated as the current training duration.

[0194] The split neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the split neural network model corresponding to the current training duration are obtained.

[0195] Based on the split neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the split neural network model corresponding to the current training duration, the neural network model is trained for a pre-specified number of rounds by using the plurality of training devices.

[0196] Optionally, it further includes a layer training duration obtaining unit, which is specifically used for: obtaining the layer training duration of each layer of the neural network model on the training device.

[0197] The initial split unit is specifically used for:

[0198] Based on the ordering of the plurality of training devices, the split information for indicating that the sample data for training the neural network model is split, and the layer training duration, the split neural network model for indicating that the neural network model is split and the sequence number information of the training device corresponding to each model segment in the split neural network model are obtained.

[0199] Optionally, the split information includes size information of the split sample data for indicating that the sample data is split.

[0200] The layer training duration obtaining unit is specifically used for:

[0201] training the neural network model a specified number of times in a model parallel manner based on the size information of the segmented sample data, to obtain a time length of forward propagation and back propagation of each layer of the neural network model, and taking the time length of forward propagation and back propagation of each layer of the neural network model as the layer training time length.

[0202] Optionally, the segmentation information used to indicate that the sample data of the neural network model is segmented includes size information of segmented sample data after the sample data is segmented and sequence number information of the segmented sample data.

[0203] The initial segmentation unit is specifically configured to:

[0204] based on the size information of the segmented sample data and the sequence number information of the segmented sample data, obtain a segmented neural network model used to indicate that the neural network model is segmented.

[0205] based on the sequence of the plurality of training devices and the segmented neural network model used to indicate that the neural network model is segmented, obtain sequence number information of a training device corresponding to each model segment in the segmented neural network model.

[0206] Optionally, the resource and parameter determination unit is specifically configured to:

[0207] determine a network topology structure diagram of the training devices used to train the neural network model by taking the training devices in the system as nodes and taking a communication bandwidth between the training devices as a weight of an edge between the nodes.

[0208] Third embodiment

[0209] Corresponding to the data processing provided in the first embodiment of the present application, the second embodiment of the present application further provides another data processing method. This embodiment is an embodiment of an application scenario of the first embodiment. In this embodiment, a user can set initial training parameters of a neural network model through a client, the client can provide the initial training parameters to a server, so that the server obtains a segmentation result that satisfies training of the neural network model, and the server finally provides the obtained segmentation result that satisfies training of the neural network model to the client. Since the method embodiment is basically similar to the first embodiment, the description is relatively simple, and the related parts are described in the part of the first embodiment. The following described embodiments are only illustrative.

[0210] The embodiment provides a data processing method, comprising:

[0211] First, obtain initial training parameters of a neural network model and training accuracy requirements set by a user through a client.

[0212] In this embodiment, the user can set the training precision requirement he wants on the client. For example, some users want to train a more precise neural network model, and some users may not have such a high requirement on the training precision. These training precision requirements can be filled in as parameters on the client, and can be passed to the server in the form of parameters through the client.

[0213] Then, the available training resources for training the neural network model are determined.

[0214] After that, the neural network model is initially segmented according to the training resources and the initial training parameters, and an initial segmentation result of the neural network model is obtained.

[0215] After obtaining the initial segmentation result, the neural network model is pre-trained based on the initial segmentation result of the neural network model, and a pre-training result is obtained.

[0216] After obtaining the pre-training result, it is determined whether the neural network model needs to be re-segmented according to the pre-training result. If so, the neural network model is re-segmented until a segmentation result that meets the training requirement of the neural network model is obtained.

[0217] Finally, the neural network model is segmented based on the segmentation result, and the segmented segments are deployed to the distributed computing nodes for training to obtain a target neural network model that meets the user's requirement. The target neural network model is returned to the client.

[0218] In this embodiment, after obtaining the segmentation result that meets the training requirement of the neural network model, the server segments the neural network model based on the segmentation result, and deploys the segmented neural network model segments to each training device for training until the neural network model meets the training precision requirement. The neural network model that meets the training precision requirement is taken as the target neural network model that meets the user's requirement, and the target neural network model is returned to the client.

[0219] Through this embodiment, users with different training precision requirements for the neural network model can set their own requirements on the client and ultimately obtain a target neural network model that meets their requirements.

[0220] Fourth embodiment

[0221] Corresponding to the method of the first embodiment of the present application, the fourth embodiment of the present application also provides an electronic device.

[0222] As shown in Figure 3 , the electronic device comprises a processor and a memory. Figure 3Fig. 4 is a schematic diagram of an electronic device according to a fourth embodiment of the present disclosure. The electronic device includes a processor 301 and a memory 302 storing a computer program which, when executed by the processor, performs the method of the first and third embodiments.

[0223] Fifth Embodiment

[0224] Corresponding to the method of the first embodiment, the fifth embodiment of the present disclosure also provides a computer storage medium storing a computer program which, when executed by a processor, performs the method of the first and third embodiments.

[0225] The present disclosure has been disclosed with reference to the preferred embodiments. However, it should be understood that the present disclosure is not limited to the preferred embodiments, but extends to any changes, alternatives or modifications within the scope and spirit of the present disclosure. Therefore, the scope of the present disclosure should be defined by the appended claims.

[0226] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. The memory can include non-persistent memory such as random access memory (RAM), non-volatile memory such as read-only memory (ROM) or flash memory, and / or a combination of such memory. The memory is an example of computer-readable media.

[0227] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include non-transitory computer-readable storage media, such as modulated data signals and carriers.

[0228] 2. Those skilled in the art will appreciate that the embodiments of the present application can be devised for as methods, systems or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system.

Claims

1. A data processing method, characterized by, The method comprises the following steps: determining available training resources and initial training parameters for training a neural network model; performing initial partitioning on the neural network model according to the training resources and the initial training parameters, to obtain an initial partitioning result of the neural network model; performing pre-training on the neural network model based on the initial partitioning result of the neural network model, to obtain a pre-training result; judging whether the number of partitions corresponding to the partitioned neural network model in the pre-training result is less than the number of layers of the neural network model; if yes, increasing the number of partitions corresponding to the partitioned neural network model in the pre-training result by 1, and performing re-partitioning on the neural network model, until a partitioning result that meets the training of the neural network model is obtained; the determination of the available training resources for training the neural network model comprises the following steps: determining a network topology structure diagram of training devices used for training the neural network model; analyzing the network topology structure diagram by using a minimum cut recursive sorting algorithm, to obtain a sorting of the training devices in the network topology structure diagram; based on the sorting of the training devices, determining available training resources of each training device used for training the neural network model.

2. The data processing method according to claim 1, characterized in that, the initial training parameters comprise partitioning information used for indicating that sample data used for training the neural network model is partitioned; the initial partitioning of the neural network model according to the training resources and the initial training parameters comprises the following steps: based on the sorting of the training devices, the partitioning information used for indicating that the sample data used for training the neural network model is partitioned, obtaining partitioned neural network models used for indicating that the neural network model is partitioned, and serial number information of the training devices corresponding to each model segment in the partitioned neural network models.

3. The data processing method according to claim 2, characterized in that, the initial training parameters comprise an initial partitioning number used for indicating that the neural network model is partitioned; the obtaining of the partitioned neural network models used for indicating that the neural network model is partitioned, and the serial number information of the training devices corresponding to each model segment in the partitioned neural network models, based on the sorting of the training devices, the partitioning information used for indicating that the sample data used for training the neural network model is partitioned, and the initial partitioning number, comprises the following steps: based on the sorting of the training devices, the partitioning information used for indicating that the sample data used for training the neural network model is partitioned, and the initial partitioning number, obtaining the partitioned neural network models used for indicating that the neural network model is partitioned, and the serial number information of the training devices corresponding to each model segment in the partitioned neural network models.

4. The data processing method according to claim 2, characterized in that, the pre-training of the neural network model based on the initial partitioning result of the neural network model comprises the following steps: based on the partitioned neural network models and the serial number information of the training devices corresponding to each model segment in the partitioned neural network models, performing pre-training on the neural network model by using the training devices for a specified number of test rounds, to obtain a current training duration; determining whether the current training duration is less than a preset minimum training duration, and if so, updating the minimum training duration as the current training duration; obtaining a segmented neural network model corresponding to the current training duration and sequence number information of a training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration; based on the segmented neural network model corresponding to the current training duration and the sequence number information of the training device corresponding to each model segment in the segmented neural network model corresponding to the current training duration, training the neural network model using the plurality of training devices for a predetermined number of rounds.

5. The data processing method according to claim 2, characterized in that, Further comprising: obtaining the layer training duration of each layer of the neural network model on the training device; based on the ordering of the plurality of training devices, the segmentation information indicating that the sample data used to train the neural network model is segmented, obtaining the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model, comprising: based on the ordering of the plurality of training devices, the segmentation information indicating that the sample data used to train the neural network model is segmented, and the layer training duration, obtaining the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model.

6. The data processing method according to claim 5, characterized in that, The segmentation information includes size information of the segmented sample data after the sample data is segmented; the layer training duration of each layer of the neural network model on the training device, comprising: based on the size information of the segmented sample data, training the neural network model in a model parallel manner for a specified number of times to obtain the duration of forward propagation and back propagation of each layer of the neural network model, and taking the duration of forward propagation and back propagation of each layer of the neural network model as the layer training duration.

7. The data processing method according to claim 2, characterized in that, The segmentation information indicating that the sample data used to train the neural network model is segmented includes size information of the segmented sample data after the sample data is segmented and sequence number information of the segmented sample data; based on the ordering of the plurality of training devices, the segmentation information indicating that the sample data used to train the neural network model is segmented, obtaining the segmented neural network model indicating that the neural network model is segmented and the sequence number information of the training device corresponding to each model segment in the segmented neural network model, comprising: based on the size information of the segmented sample data and the sequence number information of the segmented sample data, obtaining the segmented neural network model indicating that the neural network model is segmented; based on the ordering of the plurality of training devices and the segmented neural network model indicating that the neural network model is segmented, obtaining the sequence number information of the training device corresponding to each model segment in the segmented neural network model.

8. The data processing method of claim 1, wherein, The network topology diagram of the training device used to train the neural network model comprises: The training device in the system is taken as a node, and the communication bandwidth between the training devices is taken as the weight of the edge between the nodes, to determine a network topology structure diagram of the training devices for training the neural network model.

9. A data processing method, characterized by, The method comprises the steps of: obtaining initial training parameters and training accuracy requirements of a user for training a neural network model through a client; determining available training resources for training the neural network model; initially partitioning the neural network model according to the training resources and the initial training parameters to obtain an initial partitioning result of the neural network model; pre-training the neural network model based on the initial partitioning result of the neural network model to obtain a pre-training result; judging whether the number of partitions corresponding to the partitioned neural network model in the pre-training result is less than the number of layers of the neural network model; if yes, adding 1 to the number of partitions corresponding to the partitioned neural network model in the pre-training result, and re-partitioning the neural network model until a partitioning result meeting the training of the neural network model is obtained; partitioning the neural network model based on the partitioning result, deploying the partitioned segments to distributed computing nodes for training, and obtaining a target neural network model meeting the requirements of the user; returning the target neural network model to the client; the method further comprises the steps of: determining a network topology structure diagram of the training devices for training the neural network model; analyzing the network topology structure diagram by using a minimum cut recursive sorting algorithm to obtain the sorting of the multiple training devices in the network topology structure diagram; determining the available training resources of each training device among the training devices for training the neural network model based on the sorting of the multiple training devices.

10. An electronic device, comprising: The method comprises the steps of: a processor; a memory for storing a computer program, which is run by the processor to execute the method according to any one of claims 1-9.

11. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which is run by the processor to execute the method according to any one of claims 1-9.

Images