A method and apparatus for training a neural network model

Abstract

The application provides a neural network model training method and device. The neural network model training method comprises the following steps: each neural network layer in part or all of the neural network layers is divided into S parts, S accelerators included in a training system train a part of the corresponding neural network layer respectively, and each accelerator only saves a part of the training output result of each neural network layer locally. The application effectively optimizes the video memory occupied by the training output result by distributing the training output result, improves the computing power of the neural network model, and further improves the cluster training performance.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for training a neural network model. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] In the field of artificial intelligence, training neural network models yields high-performance target neural network models capable of performing various data processing tasks. When the computing power and memory size of a single accelerator are insufficient for training a neural network model, distributed parallel training can be employed. Distributed parallel training includes data-parallel training, where each accelerator in the cluster calculates the parameter gradients of the same neural network model based on different training samples. The parameter gradients obtained from each accelerator are synchronized, and the parameters of the neural network model are updated based on the synchronized gradients. If the number of accelerators in a single server is insufficient for training a neural network model, multiple accelerators from multiple servers can be used to collaboratively complete the training. However, current parallel training schemes for neural network models cannot effectively optimize the GPU memory usage of feature values, suffer from serial computation and communication, resulting in significant computational power loss, low cluster linearity, and negatively impacting cluster training performance.

[0004] Therefore, how to effectively optimize the GPU memory occupied by feature values, improve the computing power of neural network models, and thus improve the training performance of clusters has become a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a training method and apparatus for a neural network model, which can effectively optimize the video memory occupied by feature values, improve the computing power of the neural network model, and thus improve the performance of cluster training.

[0006] Firstly, a method for training a neural network model is provided. This method is applicable to a training system for a first neural network model, comprising S accelerators. The first neural network model comprises L neural network layers, where S is a positive integer greater than 1 and L is a positive integer. The method is characterized in that the first neural network layer is divided into S parts, and the first neural network layer consists of at least two of the L neural network layers. The S parts correspond one-to-one with the S accelerators, and each accelerator is used to train its corresponding part. The method may include:

[0007] The i-th accelerator in S accelerators trains the part of the 1st neural network layer in L neural network layers corresponding to the i-th accelerator, and saves the partial training output of the 1st neural network layer, i∈[1,S];

[0008] When the j-th neural network layer belongs to the first neural network layer, the i-th accelerator performs transmission processing with the S accelerators other than the i-th accelerator to obtain the first training output result. The first training output result is the training output result of the (j-1)-th neural network layer other than the training output result stored by the i-th accelerator, j∈[2,L].

[0009] The i-th accelerator trains the portion of the j-th neural network layer corresponding to the i-th accelerator based on the first training output result and the second training output result saved by the i-th accelerator, and saves the partial training output result of the j-th neural network layer. The second training output result is the partial training output result of the (j-1)-th neural network layer saved by the i-th accelerator.

[0010] It should be understood that the above method can be applied to both forward and backward training in neural network model training.

[0011] It should be understood that the first neural network layer may include at least two neural network layers, each of which is divided into S parts. This division can be done along the row direction, i.e., along the N-dimensional direction. That is, in the neural network model, each neural network layer may be divided, only a portion may be divided, or one portion may be divided along the N-dimensional direction and the other portion along the K-dimensional direction (column direction). This application does not limit this.

[0012] It should be understood that after an accelerator completes the training of a certain neural network layer, it can save the training output or a portion of the training output of that neural network layer.

[0013] It should be understood that the correspondence between accelerators and training parts can be such that when a certain accelerator, such as the first accelerator, trains the first part of the first neural network layer, or it can train other parts of the first neural network layer. When training the second neural network layer, it can train the first part of the second neural network layer, or it can train other parts of the second neural network layer, and so on. This correspondence can be predefined, and this application does not limit it.

[0014] It should also be understood that the training output can be feature values ​​or parameters such as weights.

[0015] This method effectively optimizes the GPU memory occupied by training results by distributing the training output of each layer to each accelerator, thereby improving the computing power of the neural network model and thus improving the cluster training performance.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, when the j-th neural network layer belongs to a neural network layer other than the first neural network layer among the L neural network layers, the i-th accelerator trains the part of the j-th neural network layer corresponding to the i-th accelerator according to the second training output result, and saves the training output result.

[0017] In other words, when a neural network layer is not divided according to N dimensions, the partial training output of the previous neural network layer calculated by the accelerator can be used as input parameters to calculate the part of the current neural network layer corresponding to the accelerator.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the i-th accelerator among the S accelerators trains the part of the first neural network layer of the L neural network layers corresponding to the i-th accelerator based on the training samples, and saves the training output.

[0019] That is, the input parameters for training the first neural network layer are the training samples.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the transmission processing includes global collection.

[0021] It should be understood that during forward training, the accelerator can obtain the training output results of other parts of the same neural network layer stored by other accelerators through global collection processing or global reduction processing. During backward training, the accelerator can obtain these results through global collection processing.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first accelerator belongs to a first model parallel group, and multiple accelerators in the first model parallel group train the neural network model in a model parallel manner.

[0023] In a second aspect, a training device for a neural network model is provided for training a first neural network layer model. The training system of the first neural network layer model includes S accelerators, and the first neural network layer model includes L neural network layers, where S is a positive integer greater than 1 and L is a positive integer. The first neural network layer is divided into S parts, and includes at least two neural network layers from the L neural network layers. The S parts correspond one-to-one with the S accelerators, and each accelerator is used to train its corresponding part. The training device includes a processing unit for training the part of the first neural network layer corresponding to the i-th accelerator, and storing the partial training output result of the first neural network layer, where i∈[1,S].

[0024] When the j-th neural network layer belongs to the first neural network layer, the processing unit performs transmission processing to obtain the first training output result. The first training output result is the training output result of the (j-1)-th neural network layer, excluding the part of the training output result stored by the i-th accelerator, where j∈[2,L].

[0025] The processing unit trains the portion of the j-th neural network layer corresponding to the i-th accelerator based on the first training output result and the second training output result saved by the i-th accelerator, and saves the partial training output result of the j-th neural network layer. The second training output result is the partial training output result of the (j-1)-th neural network layer saved by the i-th accelerator.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, when the j-th neural network layer belongs to a neural network layer other than the first neural network layer among the L neural network layers, the processing unit trains the part of the j-th neural network layer corresponding to the i-th accelerator according to the second training output result, and saves the partial training output result of the j-th neural network layer.

[0027] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to train the portion of the first neural network layer corresponding to the i-th accelerator in the L-th neural network layer, and to save the partial training output result of the first neural network layer.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the transmission processing includes global collection.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the training device belongs to the first model parallel group, in which multiple accelerators train the neural network model in a model parallel manner.

[0030] It should be understood that the extensions, limitations, explanations and descriptions of the relevant content in the first aspect above also apply to the same content in the second aspect.

[0031] Thirdly, a training apparatus for a neural network model is provided. The training apparatus may include a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to invoke the program instructions to execute the method described in any or all possible implementations of the first aspect.

[0032] The processor mentioned in the third aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a Google-developed application-specific integrated circuit (ASIC) for machine learning AI accelerators, but it can also be a neural network processor from Huawei; this application does not limit its scope.

[0033] Fourthly, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including methods for performing any or all possible implementations of the first aspect.

[0034] Fifthly, a chip is provided that may include a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method described in any or all possible implementations of the first aspect.

[0035] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method performed by the first accelerator in either the first aspect or the second aspect.

[0036] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0037] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method described in any or all possible implementations of the first aspect above. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0039] Figure 2 A schematic diagram of a collection communication provided in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of a data-parallel training method.

[0041] Figure 4 This is a schematic diagram illustrating a parallel training method for the model.

[0042] Figure 5 This is a schematic diagram of a hybrid parallel training method;

[0043] Figure 6 A schematic diagram of a system architecture provided for an embodiment of this application;

[0044] Figure 7 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0045] Figure 8 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0046] Figure 9 A schematic diagram of a system architecture provided for an embodiment of this application;

[0047] Figure 10 A schematic diagram of a training system provided in an embodiment of this application;

[0048] Figure 11 A schematic flowchart illustrating a training method for a neural network model provided in an embodiment of this application;

[0049] Figure 12 A schematic diagram of a neural network model provided in an embodiment of this application;

[0050] Figure 13 A schematic diagram illustrating the forward computation of a neural network model provided in an embodiment of this application;

[0051] Figure 14 A schematic diagram illustrating the reverse computation of a neural network model provided in an embodiment of this application;

[0052] Figure 15A schematic diagram of another neural network model provided in an embodiment of this application;

[0053] Figure 16 A schematic diagram illustrating the forward and backward computation of another neural network model provided in an embodiment of this application;

[0054] Figure 17 This is a schematic block diagram of a training device for a neural network model provided in an embodiment of this application;

[0055] Figure 18 This is a schematic block diagram of another training device for the neural network model provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0057] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0058] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0059] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0060] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0061] (1) Infrastructure:

[0062] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0063] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0064] The intelligent chips here can be hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0065] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0066] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0067] (2) Data:

[0068] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. This data includes graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0069] (3) Data processing:

[0070] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0071] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0072] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0073] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0074] (4) General abilities:

[0075] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0076] (5) Smart products and industry applications:

[0077] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0078] The embodiments of this application can be applied to many fields of artificial intelligence, such as intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, and safe city.

[0079] Specifically, the embodiments of this application can be applied to fields that require the use of (deep) neural networks, such as autonomous driving, image classification, image retrieval, image semantic segmentation, image quality enhancement, image super-resolution, and natural language processing.

[0080] Below is a brief introduction to two application scenarios: photo album categorization and safe city initiatives.

[0081] Photo album categories:

[0082] When users store a large number of pictures on their terminal devices (such as mobile phones) or cloud storage, recognizing the images in the album can make it easier for users or the system to classify and manage the album, thus improving the user experience.

[0083] The training method of the neural network model in this application embodiment can obtain or optimize a neural network suitable for album classification. This neural network model can then be used to classify images, thereby tagging images of different categories for easy viewing and searching by users. Furthermore, these image classification tags can be provided to the album management system for categorized management, saving users' management time, improving album management efficiency, and enhancing user experience.

[0084] Attribute recognition in the context of safe city:

[0085] In the context of safe city initiatives, various attribute recognition methods are required, such as pedestrian and cyclist attribute recognition. Deep neural networks, with their powerful capabilities, play a crucial role in these attribute recognition processes. By employing the neural network model training method described in this application, a neural network model suitable for attribute recognition in safe city scenarios can be obtained or optimized. This neural network model can then be used to process the input road images, thereby identifying different attribute information within the road images.

[0086] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0087] (1) Neural Network

[0088] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0089]

[0090] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0091] (2) Deep Neural Networks

[0092] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0093] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0094] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0095] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0096] (3) Convolutional Neural Network

[0097] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be formalized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is reducing the connections between layers in the CNN, while also reducing the risk of overfitting.

[0098] (4) Loss Function

[0099] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0100] (5) Backpropagation algorithm

[0101] Neural networks can employ backpropagation (BP) to correct the parameters of the neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0102] (6) Collective Communication Logic

[0103] The collection communication logic includes various communication methods such as all-gather, all-reduce, and reduce-scatter.

[0104] "All-gather" refers to aggregating data from multiple processes into a single process when the data is distributed across them. Multiple processes can also be understood as multiple devices.

[0105] Figure 2 In this context, process 0 (rank 0), process 1 (rank 1), process 2 (rank 2), and process 3 (rank 3) represent four different processes, i.e., four different devices; A0, A1, A2, and A3 represent the data stored on rank 0, B0, B1, B2, and B3 represent the data stored on rank 1, C0, C1, C2, and C3 represent the data stored on rank 2, and D0, D1, D2, and D3 represent the data stored on rank 3.

[0106] For example, such as Figure 2 As shown in (a), each device stores a portion of the complete data, and all-gathers data not stored locally is obtained from other devices to save the complete data locally.

[0107] All-reduce refers to processing data from multiple processes according to a specified mapping function, and storing the results in each of those processes, meaning each process stores the same result. For example, the mapping function can be a summation function, a maximum value function, a minimum value function, or an average value function.

[0108] For example, such as Figure 2 As shown in Figure (b), the specified mapping function is a summation function, which sums the data in the four processes and stores the summation result in each process.

[0109] Reduce-scatter refers to processing data from multiple processes according to a specified mapping function, and then saving different parts of the result to different processes; that is, each process saves a portion of the result. If the number of processes is N, i.e., the number of devices in the aggregated communication is N, then the amount of data saved in each process is 1 / N of the amount of data corresponding to the result. For example, the mapping function can be a summation function, a maximum value function, a minimum value function, or an average value function, etc.

[0110] For example, such as Figure 2 As shown in Figure (c), the specified mapping function is a summation function, which sums the data in the four processes and divides the summation result into four parts according to the data that each of the four processes is responsible for, and saves the four parts to the corresponding processes respectively.

[0111] (7) Distributed Machine Learning

[0112] Distributed machine learning utilizes computer clusters to enable machine learning algorithms to better train high-performance large models from big data.

[0113] Distributed machine learning includes data parallelism, model parallelism, and hybrid parallelism.

[0114] Data parallelism (DP) refers to multiple accelerators training a complete neural network model, but each accelerator only trains a portion of the training data in the dataset.

[0115] In a data-parallel approach, each device stores a copy of the model parameters, and the training data is partitioned and distributed among the devices. Each device trains the model using its assigned training data; that is, each device performs forward and backward computations based on the same model parameters to obtain parameter gradients. Once the parameter gradients obtained by each device are synchronized, the model parameters are updated. The assigned training data is typically a batch size.

[0116] For example, Figure 3 This diagram illustrates a data-parallel training method. (For example...) Figure 3 The four accelerators shown are accelerator 810, accelerator 820, accelerator 830 and accelerator 840. The complete model parameters of the neural network model are loaded on the four accelerators. The training data is divided into four batches. The four accelerators calculate the parameter gradient of the neural network model based on different batches. After synchronizing the parameter gradients calculated by each accelerator, the model parameters of the neural network model are updated based on the synchronized parameter gradients.

[0117] Model parallelism (MP) refers to training a portion of a neural network model on multiple accelerators using the full training data.

[0118] In a model-parallel approach, each device stores a subset of the model's parameters. Each device is responsible for computing different parts of the same model, collaboratively training the model. That is, each accelerator can use the same training data to perform forward and backward computations based on different model parameters, obtaining parameter gradients, and then updating the neural network model's parameters based on these gradients.

[0119] For example, Figure 4 A schematic diagram of a model parallel training method is shown. For example... Figure 4 The four accelerators shown are accelerator 910, accelerator 920, accelerator 930, and accelerator 940. The complete neural network model is divided into four parts, with each accelerator responsible for a different part. For example, different layers of the neural network model are assigned to different accelerators, or different parameters within the same layer of the neural network model are assigned to different accelerators. Figure 4 The model parameters of different parts of the neural network model are loaded onto the four accelerators, and the parameter gradients corresponding to the different parts of the model parameters are calculated based on the same training data. Then, the corresponding model parameters are updated according to the parameter gradients corresponding to the different parts of the model parameters.

[0120] Hybrid parallelism refers to combining data parallelism and model parallelism within a cluster. This addresses the challenges of large-scale training without introducing excessive communication overhead, thereby improving cluster training efficiency.

[0121] Specifically, multiple accelerators in the cluster are grouped into multiple model parallel groups. Accelerators within a group perform model parallelism, while corresponding accelerators across groups perform data parallelism. In other words, multiple accelerators within a model parallel group are each responsible for a portion of a complete neural network model, and these multiple accelerators work together to train a complete neural network model. For example... Figure 4 The four accelerators shown can be considered as a model parallel group.

[0122] For example, Figure 5 A schematic diagram of a hybrid parallel training method is shown. Figure 5 The 16 accelerators Q0, Q1...Q15 are divided into 4 groups, namely Figure 5MP groups 1, 2, 3, and 4 are configured such that the four accelerators in each group employ model parallelism, while the four groups use data parallelism across them. This means that accelerators within the same group use the same training data, while accelerators in different groups use different training data. MP groups 1 and 2 belong to server 1010, and MP groups 3 and 4 belong to server 1020. Figure 5 Each server in the system includes 8 accelerators.

[0123] The use of data parallelism among the four groups can be understood as the accelerators at corresponding positions within the four groups employing data parallelism. The accelerators at corresponding positions in different groups refer to the accelerators in different groups responsible for training the same portion of the neural network model's parameters.

[0124] For example, Q0, Q4, Q8, and Q12 are all used in four parallel model groups to train the first part of the parameters of the neural network model. Q0, Q4, Q8, and Q12 are the accelerators at the corresponding positions in the four groups. Q0, Q4, Q8, and Q12 are data parallelized, that is, Q0, Q4, Q8, and Q12 each use different training data to perform forward and backward computations based on the first part of the parameters of the neural network model to obtain parameter gradients. After synchronizing the parameter gradients obtained on Q0, Q4, Q8, and Q12, the first part of the parameters on Q0, Q4, Q8, and Q12 are updated based on the synchronized parameter gradients.

[0125] like Figure 6 As shown, this application embodiment provides a system architecture 100. In Figure 6 In this embodiment, the data acquisition device 160 is used to acquire training data. For the image processing method of this application, the training data may include training images and corresponding classification results, wherein the results of the training images may be manually pre-annotated results.

[0126] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0127] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the output image of the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0128] The aforementioned target model / rule 101 can be used to implement the image processing method of this application embodiment. Specifically, the target model / rule 101 in this application embodiment can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0129] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 6 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 6 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include the image to be processed input by the client device.

[0130] Preprocessing module 113 and preprocessing module 114 are used to preprocess the input data (such as the image to be processed) received by I / O interface 112. In this embodiment, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of them may be used), and the calculation module 111 may be used directly to process the input data.

[0131] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 may call data, code, etc., in the data storage system 150 for corresponding processing, or store the processed data, instructions, etc., into the data storage system 150. The calculation module 111 uses the target model / rule 101 to process the input data. For example, in this embodiment, the input image to be processed is processed to obtain image processing results (e.g., image classification results).

[0132] Finally, I / O interface 112 returns the processing results, such as the image classification results obtained above, to client device 140, thereby providing them to the user.

[0133] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0134] exist Figure 6 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0135] It is worth noting that, Figure 6 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 6 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0136] like Figure 6 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be the neural network in this application embodiment. Specifically, the neural network constructed in this application embodiment can be CNN, deep convolutional neural network (DCNN), recurrent neural network (RNNS), etc.

[0137] Figure 7 The present application provides a hardware structure for a chip, which includes a neural network processor 50.

[0138] This chip can be configured as follows: Figure 6 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 6 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. The algorithms for each layer in the neural network can be implemented in... Figure 7 This is achieved in the chip shown.

[0139] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0140] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0141] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0142] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0143] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0144] The unified memory 506 is used to store input data and output data.

[0145] The direct memory access controller (DMAC) stores input data from external memory into input memory 501 and / or unified memory 506, stores weight data from external memory into weight memory 502, and stores data from unified memory 506 into external memory.

[0146] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0147] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0148] The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0149] Generally, the unified memory 506, input memory 501, weight memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0150] In this embodiment of the application, the operations of each layer in the neural network can be performed by the operation circuit 503 or the vector calculation unit 507.

[0151] The training method for the neural network model in this application embodiment can be accomplished by multiple accelerators working together. Any device capable of performing operations on each layer of the neural network model can be called an accelerator. For example, an accelerator can be... Figure 7 The NPU 50 shown can be either a CPU or an accelerator.

[0152] like Figure 8 As shown in the figure, this application embodiment provides a server 400. The server 400 can be used as... Figure 6 The execution device 110 shown is used to perform the calculations of the computing module 111. The server 400 can also be used as... Figure 6 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101.

[0153] Server 400 includes a network interface card 401, a CPU 402, and multiple GPUs. For example... Figure 8 As shown, accelerators may include GPU403, GPU404, GPU405, etc. Figure 8 In the server shown, the GPU is used to perform calculations for each layer of the neural network model. Figure 8 In this context, GPU can also be understood as an accelerator. It should be understood that... Figure 8 This example uses a GPU as an accelerator; in practical applications, accelerators can also be other devices capable of performing operations on different layers of a neural network model, such as CPUs, NPUs, or TPUs. For example, an accelerator can be... Figure 7 The NPU 50 shown is not limited to this embodiment.

[0154] It should be understood that Figure 8 The number of devices shown is for illustrative purposes only; server 400 may include more than [the number of devices shown]. Figure 8 The numbers shown may be more or fewer CPUs, network cards, and GPUs. It should also be understood that... Figure 8 The architecture is only used as an example; for instance, Huawei's self-developed accelerator and network card can be integrated on a single chip. Figure 8 It is not a limitation.

[0155] Specifically, the CPU determines the computation graph and training samples that the GPU needs to perform calculations, and sends them to the GPU. The GPU then performs calculations according to the computation graph and the corresponding training samples, and feeds the calculation results back to the CPU.

[0156] For example, the training methods and communication processes of neural network models currently executed by CPUs and GPUs can be optimized through software to implement the neural network model training method in this application embodiment. For instance, distributed training software for the neural network model can be deployed on the CPU. This software generates and distributes computational graphs and training samples that require GPU computation. The accelerator performs computations according to the computational graphs and training samples and feeds back the results to the CPU. In other words, this application embodiment can be implemented in software on the basis of existing hardware. Specifically, it runs on the CPU in a server, and the CPU controls the GPU in the server to execute the neural network model training method in this application embodiment.

[0157] like Figure 9 As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local devices 301 and 302 are connected to the execution device 310 through a communication network.

[0158] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 310 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the neural network model training method of the embodiments of this application.

[0159] Specifically, in one implementation, the execution device 110 includes at least one first accelerator, which can perform the following process:

[0160] The first accelerator calculates the first sub-feature value of the first neural network layer;

[0161] The first accelerator stores the first sub-feature value locally and does not store the second sub-feature value. The first sub-feature value is a part of the first feature value, which includes the feature value of the first neural network layer. The second sub-feature value includes the part of the first feature value other than the first sub-feature value.

[0162] Through the above process, the device 110 can acquire a trained neural network, namely the target neural network, which can be used for image classification or image processing, etc.

[0163] Users can interact with execution device 310 by operating their respective user devices (e.g., local device 301 and local device 302). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0164] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0165] In one implementation, local devices 301 and 302 obtain relevant parameters of the target neural network from execution device 310, deploy the target neural network on local devices 301 and 302, and use the target neural network for image classification or image processing, etc.

[0166] In another implementation, the target neural network can be directly deployed on the execution device 310. The execution device 310 obtains the image to be processed from the local devices 301 and 302, and uses the target neural network model to classify or perform other types of image processing on the image to be processed.

[0167] It should be noted that all the functions of the execution device 310 can also be implemented by a local device. For example, local device 301 implements the functions of the execution device 310 and provides services to its own users, or provides services to the users of local device 302.

[0168] The aforementioned execution device 310 can also be a cloud device, in which case the execution device 310 can be deployed in the cloud; or, the aforementioned execution device 310 can also be a terminal device, in which case the execution device 310 can be deployed on the user terminal side. This application embodiment does not limit this.

[0169] like Figure 10 As shown in the diagram, this application provides a schematic diagram of the structure of a training system 700, which can be used as... Figure 9 The execution device 310 in the middle is used to complete the training work of the execution device to obtain the target neural network model, and can also be used as Figure 6 The training device 120 is used to complete the training work of the training device 120 and output the target model / rule 101.

[0170] The training system 700 includes servers 710, 720 and 730, which are interconnected through a communication network.

[0171] For example, a communication network can be constructed from switches. Figure 10 As shown, the communication network consists of switch 741, switch 742 and switch 743.

[0172] A server may include one or more CPUs and one or more accelerators. For example, such as Figure 10 As shown, the accelerator can be a GPU. Server 710 may include CPU 711, GPU 712, GPU 713, and GPU 714. Server 720 may include CPU 721, GPU 722, GPU 723, and GPU 724. Server 730 may include CPU 731, GPU 732, GPU 733, and GPU 734.

[0173] It should be understood that Figure 10 The number of devices shown is for illustrative purposes only; the training system 700 may include more than Figure 10 The numbers shown may be more or fewer than the number of servers, CPUs, network cards, and GPUs. It should also be understood that... Figure 10 As an example rather than a limitation, the accelerator is not limited to GPUs; it can also be other accelerators, such as Huawei's self-developed accelerator.

[0174] When the computing power and memory of a single device are insufficient for neural network training, a cluster of multiple accelerators can be used for distributed parallel training. Distributed parallel training includes data-parallel training, where each accelerator in the cluster calculates gradients for the same neural network model based on different training samples. The parameter gradients obtained from each accelerator are then synchronized, and the parameters of the neural network model are updated based on the synchronized gradients. However, these multiple accelerators may be located on multiple servers, and the low bandwidth between servers affects the data transmission speed during parameter gradient synchronization, leading to a decrease in the training efficiency of the neural network model.

[0175] In data-parallel training, the larger the model, the more memory it requires for model parameters and gradients. Each accelerator can only train the neural network model based on a small number of training samples, leading to decreased linearity and reduced overall cluster performance. When the model is too large, especially when a single accelerator cannot store all the model parameters, model parallelism can be used for training. However, model parallelism introduces high communication costs to the training cluster, further reducing its overall performance.

[0176] Hybrid parallelism can solve the problem of training large models without introducing excessive communication overhead, thus improving the training efficiency of the cluster to a certain extent.

[0177] In a hybrid parallel model, there are two possible partitioning methods. Partitioning along the first dimension allows a single accelerator to take all the feature values ​​from the previous layer as input, and then output the corresponding sub-feature values. Partitioning along the second dimension allows a single accelerator to take only a portion of the feature values ​​from the previous layer as input, and then output the corresponding feature values ​​for the current neural network layer. The first dimension can also be referred to as the N-dimensional dimension or the row direction; the second dimension can be referred to as the K-dimensional dimension or the column direction. This application does not impose any limitations on this.

[0178] However, the performance of hybrid parallelism depends on the performance of model parallelism and data parallelism. In hybrid parallel training schemes, the larger the size of the model parallel group, the greater the communication overhead brought by model parallelism. If the size of the model parallel group is too large, it will lead to excessive communication overhead, affecting the performance of model parallelism and thus the overall performance of hybrid parallelism. If the size of the model parallel group is small, the communication overhead brought by model parallelism can be reduced, ensuring the performance of model parallelism in hybrid parallelism. In other words, the size of the model parallel group can be controlled to ensure the performance of model parallelism in hybrid parallelism. However, existing model parallel algorithms can only reduce the memory usage of parameters, while the memory usage of features is not well optimized, resulting in only a small number of samples being trained per iteration. For training cluster systems with high computing power, this will lead to a significant loss of computing power on the one hand, and a decrease in the computation-to-communication ratio of the model on the other hand, thereby reducing the cluster linearity and cluster training performance. This problem is particularly obvious for large-scale cluster systems.

[0179] To address the aforementioned problems, one embodiment of this application proposes a training method 1100 for a neural network model, such as... Figure 11 As shown,

[0180] 1110: The i-th accelerator trains the portion of the 1st neural network layer corresponding to the i-th accelerator in the L neural network layers, and saves the training output results;

[0181] 1120: When the j-th neural network layer belongs to the first neural network layer, the i-th accelerator performs transmission processing with the accelerators other than the i-th accelerator among the S accelerators to obtain a first training output result. The first training output result is the training output result of the parts of the S parts of the (j-1)-th neural network layer other than the part corresponding to the i-th accelerator.

[0182] 1130: The i-th accelerator trains the part of the j-th neural network layer corresponding to the i-th accelerator according to the first training output result and the second training output result saved by the i-th accelerator, and saves the training output result. The second training output result is the training output result of the part of the (j-1)-th neural network layer corresponding to the i-th accelerator trained by the i-th accelerator.

[0183] It is worth noting that the parallel algorithm for the neural network model in this application embodiment includes a partitioning algorithm, which can have two partitioning dimensions. The partitioning dimension N (i.e., the first dimension) represents the row direction of the parameter matrix, and the partitioning dimension K (i.e., the second dimension) represents the column direction of the parameter matrix.

[0184] It should be understood that the first accelerator is not limited to a specific accelerator, but can be any accelerator in the system; similarly, the first neural network layer is not limited to a specific neural network layer, but can be any layer in the neural network.

[0185] It should be understood that the embodiments of this application can be applied to model-parallel neural network models, data-parallel neural network models, or hybrid parallel neural network models with both model and data parallelism. This application does not limit the scope of these embodiments.

[0186] It should also be understood that the training process of a neural network model basically includes forward computation and backward computation.

[0187] For the sake of clarity in describing the technical solution of this application, we will now use two GPUs to form an MP group, as follows: Figure 12 As shown, the entire neural network model is parallelized across two GPUs in an N-dimensional manner. Device refers to the device used to calculate various parameters, such as a GPU.

[0188] For ease of description, the relevant definitions are explained in advance as follows: N represents the row direction of the parameter matrix, and K represents the column direction of the parameter matrix;

[0189] W L1 D1:N(1,2)K(1,4) This refers to a submatrix of the L1 layer parameter matrix stored on Device1. The row indices of the submatrix are from 1 to 2, and the column indices are from 1 to 4. Here, the indices start from 1.

[0190] F L1 D1:N(1,2) It refers to the sub-vector of the output vector F obtained on Device1 after the forward computation of the L1 neural network. The constituent elements of this sub-vector are f1 to f2. According to the principle of matrix multiplication, the N direction of the parameter matrix, i.e. the row direction, is divided to obtain the sub-vector F. The feature is the feature value extracted from the sample after processing by the neural network.

[0191] Taking a neural network model where each layer is partitioned into N dimensions as an example, the forward computation process can be as follows: Figure 13 As shown:

[0192] S1: GPU1 calculates a portion of the L1 layer output feature, F. L1 D1:N(1,2) And it is stored locally on GPU1. GPU2 calculates another part F of the L1 layer output feature. L1 D2:N(3,4) And save it locally on GPU2;

[0193] S2: To compute the L2 layer output feature, GPU1 and GPU2 perform an all-gather operation, with GPU1 obtaining the feature from GPU2. L1 D2:N(3,4) GPU2 obtains F from GPU1 L1 D1:N(1,2) Therefore, both GPU1 and GPU2 obtained all the features of the L1 layer output;

[0194] S3: Based on the output of L1 layer, all features F L1 D1:N(1,2) and F L1 D2:N(3,4) GPU1 computes a portion of the L2 layer output feature F L2 D1:N(1,2) The data is stored locally on GPU1, and GPU2 calculates another part of the L2 layer output feature, F. L2 D2:N(3,4) And save it locally on GPU2;

[0195] S4: GPU1 only saves a portion of the L1 layer output feature. L1 D1:N(1,2) and part of the L2 layer output feature F L2 D1:N(1,2) Release F obtained from other GPUs (GPU2 in this example). L1 D2:N(3,4) GPU2 only saves a portion of the L1 layer output feature F L1 D2:N(3,4) and part of the L2 layer output feature F L2 D2:N(3,4) Release F obtained from other GPUs (GPU1 in this example). L1 D1:N(1,2) ;

[0196] And so on:

[0197] To compute the L3 layer output feature, GPU1 and GPU2 perform an all-gather operation, with GPU1 obtaining the feature from GPU2. L2 D2:N(3,4) GPU2 obtains F from GPU1 L2 D1:N(1,2) Therefore, both GPU1 and GPU2 obtained all the features F from the L2 layer output. L2 D1:N(1,2) and F L2 D2:N(3,4) ;

[0198] S5: All features based on the output of the L2 layer (F) L2 D1:N(1,2) and F L2 D2:N(3,4) GPU1 computes a portion of the L3 layer output feature, F. L3 D1:N(1,2) And it is stored locally on GPU1. GPU2 calculates another part F of the L3 layer output feature. L3 D2:N(3,4) And save it locally on GPU2;

[0199] S6: GPU1 saves a portion of the L3 layer output feature. L3 D1:N(1,2) Release F obtained from GPU2 L2 D2:N(3,4) GPU2 saves a portion of the L3 layer output feature F L3 D2:N(3,4) Release F obtained from GPU1 L2 D1:N(1,2) ;

[0200] This process continues until all forward layers have been computed. At this point, GPU1 stores a portion of the output feature for each layer: F L1 D1:N(1,2) F L2 D1:N(1,2) F L3 D1:N(1,2) …GPU2 stores another part of the output feature for each layer: F L1 D2:N(3,4) F L2 D2:N(3,4) F L3 D2:N(3,4) …

[0201] Taking a neural network model where each layer of the neural network is split in N dimensions as an example, for a neural network layer that is split in N dimensions, an all-gather operation is needed before the reverse computation of that layer to obtain the complete input feature of that layer (the output feature of the previous layer).

[0202] In addition, a more optimized approach is to synchronize the all-gather transmission of each layer's input feature with the reverse computation of the layer following it (overlap), thereby reducing the impact of additional transmission on system performance.

[0203] This embodiment illustrates the method of optimizing data flow through reverse computation, as follows: Figure 14 The following is stated:

[0204] S1: In order to reverse compute the L4 layer, both GPU1 and GPU2 need to obtain all the features F output from the L3 layer. L3 D1:N(1,2) and F L3 D2:N(3,4) A gather operation is performed between GPU1 and GPU2, and GPU1 obtains a portion of the L3 layer output feature F from GPU2. L3 D2:N(3,4) GPU2 obtains a portion of the L3 layer output feature F from GPU1. L3 D1:N(1,2) ;

[0205] S2: GPU1 and GPU2 perform reverse computation of the L4 layer. Simultaneously, GPU1 and GPU2 perform an all-gather operation, and GPU1 obtains a partial feature F from the L2 layer output of GPU2. L2 D2:N(3,4) GPU2 obtains a portion of the L2 layer output feature F from GPU1. L2 D1:N(1,2) ;

[0206] S3: After GPU1 and GPU2 complete the reverse computation of the L4 layer, release all features of the L3 layer. L3 D1:N(1,2) and F L3 D2:N(3,4) ;

[0207] And so on:

[0208] S4: GPU1 and GPU2 perform reverse computation of the L3 layer. Simultaneously, GPU1 and GPU2 perform an all-gather operation, and GPU1 obtains a partial feature F from the L1 layer output of GPU2. L1 D2:N(3,4)GPU2 obtains a portion of the L1 layer output feature F from GPU1. L1 D1:N(1,2) ;

[0209] S5: After GPU1 and GPU2 complete the reverse computation of the L3 layer, release all features of the L2 layer. L2 D1:N(1,2) and featureF L2 D2:N(3,4) ;

[0210] This process continues until all layers have completed the reverse calculation.

[0211] In this embodiment, the features are stored in a distributed manner (within the parallel model group), which reduces the memory usage of each device and increases the number of samples for a single training iteration, thus significantly improving the training performance of the AI ​​cluster.

[0212] Another embodiment of this application, taking the NVIDIA neural network model as an example, provides a method for training a neural network model:

[0213] like Figure 15 As shown, taking an MP group consisting of two GPUs as an example, the entire neural network model is parallelized between the two GPUs using NVIDIA model parallelism. The L1 layer is split into N dimensions, the L2 layer into K dimensions, the L3 layer into N dimensions, the L4 layer into K dimensions, and so on.

[0214] Specifically, the detailed process of forward computation is described as follows: Figure 16 As shown in (a):

[0215] S1: GPU1 calculates a portion of the L1 layer output feature, F. L1 D1:N(1,2) And it is stored locally on GPU1. GPU2 calculates another part F of the L1 layer output feature. L1 D2:N(3,4) And save it locally on GPU2; GPU1 is based on the locally saved F L1 D1:N(1,2) Calculate the L2 layer output feature F L2 D2:N(3,4) GPU2 is based on locally stored F L1 D2:N(3,4) Calculate the L2 layer output feature.

[0216] S2: To compute the L3 layer output feature, GPU1 and GPU2 exchange L2 layer output feature F. L2 D1:N(1,2) and FL2 D2:N(3,4) Perform all-reduce calculations, and use the results of the all-reduce calculations as the total features output by the L2 layer.

[0217] S3: Based on all features output from the L2 layer in step S2, GPU1 computes a portion of the L3 layer output features, F. L3 D1:N(1,2) And it is stored locally on GPU1, while GPU2 computes another part of the L3 layer output feature, F. L3 D1:N(1,2) And save it locally on GPU2; GPU1 saves a portion of the total features output from the L2 layer. L2 D1:N(1,2) Release the other parts, GPU2 saves another part of the L2 layer output feature F. L2 D2:N(3,4) Release the other parts;

[0218] S4: GPU1 is based on locally saved feature F L3 D1:N(1,2) To compute the L4 layer output, GPU2 uses locally stored F... L3 D2:N(3,4) Calculate the output feature of the L4 layer.

[0219] This process continues until all forward layers have been computed. At this point, GPU1 stores a portion of the output feature F from each layer. L1 D1:N(1,2) F L2 D1:N(1,2) F L3 D1:N(1,2) …GPU2 stores another part of the output feature for each layer: F L1 D2:N(3,4) F L2 D2:N(3,4) F L3 D2:N(3,4) …

[0220] For a neural network layer that performs N-dimensional splitting, an all-gather operation is needed before back-computing to obtain the complete input feature of the layer (the output feature of the previous layer).

[0221] In addition, a more optimized approach is to synchronize the all-gather transmission of each layer's input feature with the reverse computation of the layer following it, thereby reducing the impact of additional transmission on system performance.

[0222] This embodiment illustrates the method of optimizing data flow through reverse computation, as follows: Figure 16 As shown in (b):

[0223] S1: GPU1 outputs L5 layer features based on locally stored data. L5 D1:N(1,2) Perform L6 layer reverse computation; GPU2 uses locally stored L5 layer output feature F. L5 D2:N(3,4) Perform inverse computation of the L6 layer; simultaneously, perform an allgather operation between GPU1 and GPU2, with GPU1 obtaining a portion of the L4 layer output feature F from GPU2. L4 D1:N(1,2) GPU2 obtains a portion of the L4 layer output feature F from GPU1. L4 D2:N(3,4) ;

[0224] S2: After GPU1 and GPU2 complete the reverse computation of L6 layer, release all features of L5 layer F. L5 D1:N(1,2) and F L5 D2:N(3,4) ;

[0225] S3: GPU1 and GPU2 perform reverse computation of L5 layer. After the computation is complete, all features of L4 layer F are released. L4 D1:N(1,2) and F L4 D2:N(3,4) .

[0226] And so on:

[0227] S4: GPU1 outputs L3 layer feature F based on locally stored data. L3 D1:N(1,2) Perform L4 layer reverse computation; GPU2 uses locally stored L3 layer output feature F. L3 D2:N(3,4) Perform inverse computation of the L4 layer; simultaneously, perform an allgather operation between GPU1 and GPU2, with GPU1 obtaining a portion of the L2 layer output feature F from GPU2. L2 D2:N(3,4) GPU2 obtains a portion of the L2 layer output feature F from GPU1. L2 D1:N(1,2) ;

[0228] S5: After GPU1 and GPU2 complete the reverse computation of L4 layer, release all features of L3 layer F. L3 D1:N(1,2) and F L3D2:N(3,4) ;

[0229] S6: GPU1 and GPU2 perform reverse computation of the L3 layer. After the computation is complete, all features of the L2 layer are released. L2 D1:N(1,2) and F L2 D2:N(3,4) .

[0230] This process continues until all layers have completed the reverse calculation.

[0231] It should be understood that the feature values ​​of the previous layer can be obtained during reverse training either before the current training or during the reverse training of the previous layer, such as... Figure 16 As shown in (c) in the document, this application does not limit this.

[0232] This application uses distributed storage for features (within the parallel model group), which reduces the memory footprint of each device and increases the number of samples per training iteration, thus significantly improving the training performance of AI clusters.

[0233] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0234] In the embodiments provided above, the methods provided by the embodiments of this application have been described from the perspective of interaction between various devices. To implement the functions of the methods provided in the embodiments of this application, network devices or terminal devices may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0235] The module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0236] The following is combined Figures 17 to 18 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0237] Figure 17 This is a schematic block diagram of a training apparatus for a neural network model according to an embodiment of this application. Figure 17 The training device 3000 shown includes an acquisition unit 3010 and a processing unit 3020.

[0238] The acquisition unit 3010 and the processing unit 3020 can be used to execute the training method of the neural network model of the present application embodiment, specifically, they can be used to execute method 1100.

[0239] The processing unit 3020 is used to calculate the first sub-feature value of the first neural network layer, store the first sub-feature value locally, and not store the second sub-feature value. The first sub-feature value is a part of the first feature value, the first feature value includes the feature value of the first neural network layer, and the second sub-feature value includes the part of the first feature value other than the first sub-feature value.

[0240] Optionally, as an embodiment, the training device belongs to a first model parallel group, in which multiple accelerators train the neural network model in a model parallel manner.

[0241] The above is just an example. Processing unit 3020 can also perform other functions. For a more detailed description, please refer to [link / reference needed]. Figures 11 to 16 The descriptions of the method embodiments shown or other method embodiments are not repeated here.

[0242] It should be noted that the aforementioned training device 3000 is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, without specific limitations.

[0243] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0244] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] Figure 18This is a schematic diagram of the hardware structure of the training device for the neural network model provided in this application embodiment. Figure 18 The training device 5000 shown (which may specifically be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0246] The memory 5001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 can store programs. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 executes the various steps of the neural network model training method of this application embodiment. Specifically, the processor 5002 can execute the steps described above... Figure 11 Step S1120 in the method shown.

[0247] The processor 5002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the neural network model training method of the method embodiment of this application.

[0248] The processor 5002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 7 The chip shown. In the implementation process, each step of the training method for the neural network model of this application can be completed by the integrated logic circuit of the hardware in the processor 5002 or by software instructions.

[0249] The processor 5002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 5001, and the processor 5002 reads the information in memory 5001 and combines it with its hardware to complete the task. Figure 17 The training apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figure 11 The training method for the neural network model shown.

[0250] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, training samples or parameter gradients can be obtained through the communication interface 5003.

[0251] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0252] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0253] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0254] When the aforementioned communication device is a chip applied to a neural network, the chip implements the functions of the neural network in the above method embodiments.

[0255] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0256] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0257] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0258] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0259] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0260] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0261] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0262] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0263] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0264] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0265] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training a neural network model, used to train a first neural network model, wherein the training system of the first neural network model includes S accelerators, the first neural network model includes L neural network layers, S is a positive integer greater than 1, and L is a positive integer, characterized in that, in, Each neural network layer in the first neural network layer is divided into S parts. The first neural network layer includes at least two neural network layers from the L neural network layers. The S parts correspond one-to-one with the S accelerators. Each accelerator is used to train the corresponding part, including: The i-th accelerator of the S accelerators trains the part of the 1st neural network layer of the L neural network layers corresponding to the i-th accelerator, and saves the partial training output of the 1st neural network layer, i∈[1,S]; When the j-th neural network layer belongs to the first neural network layer, the i-th accelerator performs transmission processing with the S accelerators other than the i-th accelerator to obtain the first training output result. The first training output result is the training output result of the (j-1)-th neural network layer other than the training output result stored by the i-th accelerator, j∈[2,L]. The i-th accelerator trains the portion of the j-th neural network layer corresponding to the i-th accelerator based on the first training output result and the second training output result saved by the i-th accelerator, and saves the partial training output result of the j-th neural network layer. The second training output result is the partial training output result of the (j-1)-th neural network layer saved by the i-th accelerator. The i-th accelerator belongs to the first model parallel group, and multiple accelerators in the first model parallel group train the neural network model in a model parallel manner.

2. The method according to claim 1, characterized in that, The method further includes: When the j-th neural network layer belongs to a neural network layer other than the first neural network layer among the L neural network layers, the i-th accelerator trains the part of the j-th neural network layer corresponding to the i-th accelerator according to the second training output result, and saves the partial training output result of the j-th neural network layer.

3. The method according to claim 1 or 2, characterized in that, The i-th accelerator of the S accelerators trains the portion of the 1st neural network layer of the L neural network layers corresponding to the i-th accelerator, and saves the partial training output of the 1st neural network layer, i∈[1,S], including: The i-th accelerator in the S accelerators trains the portion of the 1st neural network layer of the L neural network layers corresponding to the i-th accelerator based on the training samples, and saves the partial training output of the 1st neural network layer.

4. The method according to claim 1 or 2, characterized in that, The transmission process includes global collection.

5. The method according to claim 1 or 2, characterized in that, When the training is reverse training, the transmission processing also includes global reduction.

6. A training apparatus for a neural network model, used to train a first neural network model, wherein the training system of the first neural network model includes S accelerators, the first neural network model includes L neural network layers, S is a positive integer greater than 1, and L is a positive integer, characterized in that, in, The first neural network layer is divided into S parts, and the first neural network layer is at least two neural network layers among the L neural network layers. The S parts correspond one-to-one with the S accelerators, and each accelerator is used to train the corresponding part. The training device is used as the i-th accelerator among the S accelerators, and the training device includes: The processing unit is used to train the part of the first neural network layer corresponding to the i-th accelerator in the L neural network layers, and save the partial training output of the first neural network layer, i∈[1,S]; When the j-th neural network layer belongs to the first neural network layer, the processing unit performs transmission processing to obtain the first training output result. The first training output result is the training output result of the (j-1)-th neural network layer, excluding the training output result stored by the i-th accelerator, where j∈[2,L]. The processing unit trains the portion of the j-th neural network layer corresponding to the i-th accelerator based on the first training output result and the second training output result saved by the i-th accelerator, and saves the partial training output result of the j-th neural network layer. The second training output result is the partial training output result of the (j-1)-th neural network layer saved by the i-th accelerator. The i-th accelerator belongs to the first model parallel group, and multiple accelerators in the first model parallel group train the neural network model in a model parallel manner.

7. The training device according to claim 6, characterized in that, When the j-th neural network layer belongs to a neural network layer other than the first neural network layer among the L neural network layers, the processing unit trains the part of the j-th neural network layer corresponding to the i-th accelerator according to the second training output result, and saves the partial training output result of the j-th neural network layer.

8. The training device according to claim 6 or 7, characterized in that, The processing unit is specifically used to train the portion of the first neural network layer corresponding to the i-th accelerator in the L neural network layers, and to save the partial training output results of the first neural network layer.

9. The training device according to claim 6 or 7, characterized in that, The transmission process includes global collection.

10. The training device according to claim 6 or 7, characterized in that, When the training is reverse training, the transmission processing also includes global reduction.

11. A training device for a neural network model, characterized in that, It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions to perform the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including methods for performing any one of claims 1 to 5.

13. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the method as described in any one of claims 1 to 5.

Images