Method and device for multiplexing memory of neural network chip

Abstract

The invention relates to the field of AI, and provides a method and device for multiplexing a neural network chip memory. The method comprises the steps of determining a target tensor, wherein the target tensor is a tensor needing to be exchanged from a neural network chip memory to a non-neural network chip memory; the life cycle of the target tensor is determined according to the multiple calculation operations participated by the target tensor, the life cycle of the target tensor is larger than or equal to the sum of the durations of the multiple calculation operations, and the life cycle of the target tensor is smaller than the duration from the starting moment of the multiple calculation operations to the ending moment of the multiple calculation operations; and multiplexing the memory of the neural network chip according to the life cycle of the target tensor. According to the method, the life cycle of the target tensor is redefined, and the idle neural network chip memory is allocated to the tensor participating in calculation operation, so that the use efficiency of the neural network chip memory is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a method and device for reusing the memory of a neural network chip. Background Art

[0002] In the field of AI, the "big data + big model" computing method under unsupervised learning conditions is one of the important paths to "general artificial intelligence". AI big models have strong learning and understanding capabilities, and have performed well in natural language processing, machine translation, text generation, dialogue systems, and image processing. In recent years, the number of parameters of AI big models has become larger and larger, showing an exponential growth trend. The number of parameters has exceeded the 100 billion level and is developing towards the trillion scale. Such a large number of parameters makes the calculation (i.e., training / inference) of AI big models face a serious memory wall problem, because a neural network chip usually has only tens of gigabytes (GB) of memory space. Studies have shown that training AI big models with hundreds of billions of parameters requires more than 400 neural network chips (80GB memory each). So how to efficiently use the memory of neural network chips has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application provide a method, device, computer-readable storage medium, and computer program product for reusing the memory of a neural network chip, which can improve the utilization efficiency of the memory of a neural network chip.

[0004] In the first aspect, a method for reusing the memory of a neural network chip is provided. The execution subject of the method can be a neural network computing device or a processor applied to the neural network computing device. The following description is made by taking the execution subject as a neural network computing device (referred to as "computing device") as an example. The method includes: determining a target tensor, which is a tensor that needs to be exchanged from the neural network chip memory to the non-neural network chip memory; determining the life cycle of the target tensor according to multiple computing operations in which the target tensor participates, wherein the life cycle of the target tensor is greater than or equal to the sum of the durations of the multiple computing operations, and the life cycle of the target tensor is less than the duration from the start time of the multiple computing operations to the end time of the multiple computing operations; reusing the neural network chip memory according to the life cycle of the target tensor.

[0005] In the prior art, the life cycle of a tensor lasts from the first computing operation it participates in to the last computing operation it participates in. During this period of time, even if the tensor does not participate in a computing operation, it cannot be exchanged from the neural network chip memory to the non-neural network chip memory. In this embodiment, the life cycle of the target tensor is less than the duration from the start time of multiple computing operations to the end time of multiple computing operations. During the interval between these multiple computing operations, the target tensor does not participate in the computing operation. The computing device can exchange the target tensor from the neural network chip memory to the non-neural network chip memory, and by redefining the life cycle of the target tensor, the idle neural network chip memory is allocated to the tensor participating in the computing operation, thereby improving the use efficiency of the neural network chip memory.

[0006] Optionally, the target tensor is a tensor that cannot be generated multiple times during the computation of the neural network.

[0007] Some tensors can be generated multiple times through computing operations. When such tensors need to be used, they can be directly generated through computing operations without performing exchange operations on such tensors, thus saving the exchange overhead.

[0008] Optionally, the method also includes: determining multiple tensors, all of which are tensors that need to be exchanged from the neural network chip memory to the non-neural network chip memory, and the multiple tensors include a target tensor; reusing the neural network chip memory according to the life cycle of the target tensor, including: according to the life cycle of the multiple tensors, exchanging the multiple tensors from the neural network chip memory to the non-neural network chip memory through a single exchange operation; and / or, according to the life cycle of the multiple tensors, exchanging the multiple tensors from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

[0009] The swap operation generates data movement overhead. If each tensor to be swapped performs a swap operation, a large amount of data movement overhead will be generated. This embodiment performs a swap operation on multiple tensors to be swapped, saving the data movement overhead caused by the swap operation.

[0010] Optionally, the neural network chip memory is reused according to the life cycle of the target tensor, including: according to the life cycle of multiple tensors, multiple tensors are exchanged from the neural network chip memory to the non-neural network chip memory through a single exchange operation; the multiple tensors include a first tensor participating in the forward computing operation of the first neural network layer; determining multiple tensors, including: determining the first tensor before the forward computing operation of the first neural network layer is completed; after the forward computing operation of the first neural network layer is completed, and before the first tensor is exchanged, the method also includes: executing the forward computing operation of the second neural network layer, the second neural network layer being a neural network layer located after the first neural network layer.

[0011] In this embodiment, the first tensor is a tensor that participates in the forward computing operation of the first neural network layer and needs to be exchanged to the non-neural network chip memory. The first tensor is determined before the forward computing operation of the first neural network layer ends, and is exchanged to the non-neural network chip memory during the forward computing operation of the second neural network layer. In this way, the forward computing operation of the second neural network layer does not need to wait for the exchange operation of the first tensor, thereby reducing the negative impact of the exchange operation on the training speed.

[0012] Optionally, the neural network chip memory is reused according to the life cycle of the target tensor, including: according to the life cycle of multiple tensors, multiple tensors are exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation; the multiple tensors include a first tensor participating in the reverse computing operation of the first neural network layer; multiple tensors are determined, including: before the reverse computing operation of the second neural network layer is completed, the second neural network layer is a neural network layer located after the first neural network layer; multiple tensors are exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation, including: before the reverse computing operation of the second neural network layer is completed, the first tensor is exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

[0013] In this embodiment, the first tensor is a tensor that participates in the reverse computing operation of the first neural network layer and needs to be exchanged to the neural network chip memory. The first tensor is determined before the reverse computing operation of the second neural network layer is completed, and is exchanged to the non-neural network chip memory before the reverse computing operation of the second neural network layer is completed. In this way, the exchange operation of the first tensor and the reverse computing operation of the second neural network layer can be performed simultaneously, and the reverse computing operation of the first neural network layer does not need to wait for the exchange operation of the first tensor, thereby reducing the negative impact of the exchange operation on the training speed.

[0014] Optionally, the non-neural network chip memory includes a first memory and a second memory, and the transmission rate of the first memory is higher than the transmission rate of the second memory; exchanging multiple tensors from the neural network chip memory to the non-neural network chip memory includes: exchanging tensors with shorter calculation intervals among the multiple tensors to the first memory; exchanging tensors with longer calculation intervals among the multiple tensors to the second memory; wherein the calculation interval is the interval time between the first time any tensor among the multiple tensors participates in a calculation operation and the last time it participates in a calculation operation.

[0015] The longer the calculation interval of the tensor, the longer the time available for the exchange operation, and a lower transmission rate can also be used to exchange the tensor to the neural network chip memory before the calculation operation is performed. The shorter the calculation interval of the tensor, the shorter the time available for the exchange operation, and a higher transmission rate is required to exchange the tensor to the neural network chip memory before the calculation operation is performed. In this embodiment, tensors with longer calculation intervals are exchanged to memory with lower transmission rates, and tensors with shorter calculation intervals are exchanged to memory with higher transmission rates. This can reasonably utilize non-neural network chip memory to meet the exchange requirements of different tensors.

[0016] Optionally, the tensor with a shorter calculation interval is the tensor corresponding to the neural network layer after the target neural network layer, and the tensor with a longer calculation interval is the tensor corresponding to the target neural network layer and the neural network layer before the target neural network layer.

[0017] The later the neural network layer is, the smaller the calculation interval between the forward calculation operation and the reverse calculation operation is, and the tensors to be exchanged corresponding to these neural network layers need to use memory with a higher transmission rate. The earlier the neural network layer is, the larger the calculation interval between the forward calculation operation and the reverse calculation operation is, and the tensors to be exchanged corresponding to these neural network layers can use memory with a lower transmission rate. Therefore, this embodiment can reasonably utilize the non-neural network chip memory to meet the exchange requirements of different tensors.

[0018] Optionally, the number of neural network layers after the target neural network layer is equal to the smaller of the first value and the number of neural network layers, and the first value is the result of rounding down the ratio of the capacity of the first memory to the amount of data of the tensor to be exchanged corresponding to the single-layer neural network.

[0019] The larger the first value is, the larger the capacity of the first memory is. For example, if the first value is 3, it means that the first memory can carry the tensors to be exchanged of three neural network layers. If the number of neural network layers is 2, the tensors to be exchanged of all neural network layers can be exchanged to the first memory; if the number of neural network layers is 6, the tensors to be exchanged of some neural network layers (3 layers) can be exchanged to the first memory. Therefore, this embodiment can make full use of the high transmission rate of the first memory, while avoiding exchange errors caused by excessive data volume of the tensors to be exchanged.

[0020] In a second aspect, an embodiment of the present application provides a device for reusing the memory of a neural network chip. The device may include a processing unit for executing: any method in the above-mentioned first aspect and its optional implementation manner.

[0021] In a third aspect, an embodiment of the present application provides a device for reusing the memory of a neural network chip, which may be an electronic device or a chip applied to an electronic device. The device may include a processor for executing: any method in the first aspect and its optional implementations.

[0022] Optionally, the device may further include a transceiver. When the device is an electronic device, the transceiver may be a transceiver circuit, an antenna, etc.; when the device is a chip applied to an electronic device, the transceiver may be an input / output interface, a pin, a circuit, etc.

[0023] Optionally, the device may further include a memory, the memory being used to store instructions, and the processor executing the instructions stored in the memory so that the device performs any one of the methods in the first aspect and its optional implementations. When the device is an electronic device, the memory may be a read-only memory, a random access memory, etc.; when the device is a chip applied to an electronic device, the memory may be a register, a cache, etc.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed on a computer, the computer executes any method in the first aspect and its optional embodiments.

[0025] In the fifth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code or computer program instructions. When the computer program code or computer program instructions are executed by a device that reuses the memory of a neural network chip, the device executes any one of the methods in the first aspect and its optional embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram of a neural network provided in an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of a method for training a neural network provided in an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a calculation graph provided in an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of a memory reuse method provided in an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of a switching operation provided in an embodiment of the present application;

[0031] Figure 6is a schematic diagram of a memory usage change provided by an embodiment of the present application;

[0032] Figure 7 is a schematic diagram of a neural network computing system provided in an embodiment of the present application;

[0033] Figure 8 It is a schematic diagram of an AI large model software protocol stack provided in an embodiment of the present application;

[0034] Fig. 9 is a schematic diagram of a method for reusing a neural network chip memory provided in an embodiment of the present application;

[0035] Fig.10 is a schematic diagram of a recalculation method provided in an embodiment of the present application;

[0036] Fig.11 is a schematic diagram of a redefinition method provided in an embodiment of the present application;

[0037] Fig.12 is a schematic diagram of another redefinition method provided in an embodiment of the present application;

[0038] Fig.13 It is a schematic diagram of the change of memory usage after an application redefinition method and a memory reuse method provided in an embodiment of the present application;

[0039] Fig.14 is a schematic diagram of an asynchronous batch exchange method provided in an embodiment of the present application;

[0040] Fig.15 It is a schematic diagram of a memory hierarchical switching method provided in an embodiment of the present application;

[0041] Fig.16 is a schematic diagram of another method for reusing the memory of a neural network chip provided in an embodiment of the present application;

[0042] Fig.17 This is a schematic diagram of the effect of applying the method for reusing the memory of a neural network chip provided in an embodiment of the present application;

[0043] Fig.18 is a schematic diagram of a device for reusing neural network chip memory provided in an embodiment of the present application;

[0044] Fig.19 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0046] To facilitate understanding of the technical solutions of the embodiments of the present application, a brief introduction to the concepts involved in the embodiments of the present application is first given.

[0047] (1) Neural network (NN).

[0048] Neural network is also called artificial neural network (ANN) or quasi-neural network. In the field of machine learning and cognitive science, it is a mathematical model or computational model that imitates the structure and function of biological neural network and is used to estimate and approximate functions. Neural network can include multilayer perceptron (MLP), convolutional neural network (CNN) and deep neural network (DNN).

[0049] A neural network can include multiple neural network layers, such as Figure 1 As shown, the neural network 100 may include m neural network layers, where m is a positive integer, wherein the first layer is the input layer, the mth layer is the output layer, and the remaining layers are hidden layers, and the arrows between the layers indicate the direction of data transmission. In an embodiment of the present application, a neural network layer is a logical probability, and a neural network layer may refer to a set of computing nodes corresponding to a neural network calculation, and a computing node may be referred to as a neuron. In practical applications, a neural network layer may include a convolutional layer, a pooling layer, a fully connected layer, and the like.

[0050] The working principle of each neural network layer can be expressed mathematically To describe, among them, represents a sample (i.e., input data). A sample generally has multiple attributes, so a sample is usually a vector; w represents the weight value of each neuron in the neural network; b is the bias; a is the activation function; is the output vector. From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., a set of output vectors) through five operations on the input space (i.e., a set of input vectors), where: Determines the transformation from input space to output space, that is, w of each layer controls how to transform the space. The above five operations include: 1. Dimensional increase / decrease; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2 and 3 are composed of Operation 4 is implemented by +b, and operation 5 is implemented by a().

[0051] (2) Training of neural networks.

[0052] Because we want the output of the neural network to be as close to the actual situation as possible, we can determine the difference between the predicted value and the target value of the neural network by comparing them, and then update the parameters of the neural network (such as weight value) based on the difference between the two. There is usually an initialization process before the first parameter update, that is, pre-configuring parameters for each layer of the neural network. During the training process, if the predicted value is high, adjust the parameters to reduce the predicted value of the neural network; if the predicted value is low, adjust the parameters to increase the predicted value of the neural network; until the output of the neural network meets the requirements, therefore, it is necessary to pre-define how to compare the difference between the predicted value and the target value. The loss function and the objective function are commonly used means to measure the difference between the predicted value and the target value. Taking the loss function as an example, the larger the output value (loss) of the loss function, the greater the difference; the smaller the output value of the loss function, the smaller the difference. Then, the training of the neural network becomes a process of minimizing this output value as much as possible.

[0053] like Figure 2 As shown in the figure, the starting point corresponds to the output value of the loss function when the neural network parameters are not updated. As the neural network parameters are updated, the output value of the loss function gradually decreases. The optimal point corresponds to the minimum output value of the loss function. When the output value of the loss function reaches the minimum value, the training of the neural network is completed.

[0054] The loss function is usually a multivariable function. The gradient can reflect the rate of change of the output value of the loss function when the variable changes. The larger the absolute value of the gradient, the greater the rate of change of the output value of the loss function. The gradient of the loss function when updating different parameters can be calculated, and the parameters can be updated in the direction where the gradient descent is fastest to reduce the output value of the loss function as quickly as possible.

[0055] The output value of the loss function of the neural network can be obtained through forward propagation (FP) calculation. For example, the output result of the previous layer of the neural network can be input into the next layer until the output result of the output layer is obtained, and the output value of the loss function is calculated based on the output result and the target value. Subsequently, a back propagation (BP) calculation is performed based on the output value of the loss function to obtain the gradient of each layer, and the sum is adjusted along the direction of the fastest gradient descent until the output value of the loss function reaches the minimum value. The above forward propagation calculation can also be called forward calculation, and the above back propagation calculation can also be called reverse calculation.

[0056] (3) Computational graph.

[0057] Neural networks usually have the characteristics of static calculation, that is, after the model parameters of the neural network are determined, the calculation mode and required memory size of the neural network can be determined during the graph compilation period. Based on this feature, the calculation of the neural network can be abstracted into the form of a calculation graph.

[0058] Figure 3 This is an example of a calculation graph provided in an embodiment of the present application.

[0059] Figure 3 In the calculation diagram shown, each rectangle represents a calculation operation, each circle represents a data, and each arrow represents the flow of data. In the calculation process of the neural network, the training data and intermediate results can be regarded as tensors. The concept of tensor includes scalar, vector and linear operator, which can be represented by n-dimensional array, where n is a positive integer.

[0060] Figure 3 The calculation operations shown include a multiplication operator (Mul), a data type conversion operator (Cast), an addition operator (Add), a division operator (Div), and a subtraction operator (Sub). The multiplication operator performs calculation operations on tensors 1 and 2 and outputs tensor 4, and the data type conversion operator performs calculation operations on tensor 3 and outputs tensor 5. Tensors 4 and 5 are input into the addition operator, and the addition operator outputs tensor 6 after calculation operations. Tensors 5 and 6 are input into the division operator, and the division operator outputs tensor 7 after calculation operations. Finally, the subtraction operator performs calculation operations on tensors 6 and 7 and outputs tensor 8, which is the output tensor of the neural network.

[0061] The above operators are the model parameters (i.e., weights) of the neural network. In each iterative calculation, the model parameters participate in the forward and reverse calculations of the neural network and reside in the memory. Therefore, the memory occupied by the model parameters is called static memory. Tensors 4, 5, 6, and 7 are intermediate tensors generated by the neural network during the training / inference stage. After one training / inference, the intermediate tensors are no longer used and these four tensors will be deleted from the memory. Therefore, the memory occupied by these four tensors is called dynamic memory.

[0062] (4)Memory reuse.

[0063] The period of time that a tensor participates in a computation is called the tensor lifecycle, which is represented by Figure 3 It can be seen that the computational operations of neural networks have a sequence, so tensors with non-overlapping life cycles can use the same memory space without conflict.

[0064] Figure 4 This is an example of a memory reuse method provided in an embodiment of the present application.

[0065] Figure 4In the figure, rectangles a to j represent ten tensors that need to use memory space. The length of the rectangle corresponds to the life cycle of the tensor, and the width of the rectangle corresponds to the size of the memory space occupied by the tensor. For example, the length of tensor a is 2 units, which means that the life cycle of tensor a is 2 time units, and the width of tensor a is 5 units, which means that the memory space occupied by tensor a is 5 memory units; the length of tensor b is 8 units, which means that the life cycle of tensor b is 8 time units, and the width of tensor a is 5 units, which means that the memory space occupied by tensor b is 5 memory units.

[0066] The life cycles of tensor a, tensor b, and tensor f overlap. Therefore, tensor a, tensor b, and tensor f cannot use the same memory space at the same time. The life cycles of tensor a and tensor c, tensor a and tensor d, tensor a and tensor e, tensor a and tensor g, tensor a and tensor h, tensor a and tensor i, and tensor a and tensor j do not overlap. Therefore, tensor a can use the same memory space with tensor c, tensor d, tensor e, tensor g, tensor h, tensor i, and tensor j respectively.

[0067] The situation where different tensors use the same memory segment at the same time is called memory reuse. When memory reuse is not performed, the memory usage of tensors a~j is as follows Figure 4 As shown in the left part of , a total of 37 units of memory space are occupied. After memory reuse is performed, the memory usage of tensors a~j is as follows Figure 4 As shown in the right part of , a total of 24 units of memory space are occupied. This shows that memory reuse compresses the memory space occupied by tensors.

[0068] In the above memory management method, memory allocation is determined according to the reuse strategy during the compilation phase of the neural network, and no memory allocation is performed during the calculation phase of the neural network. Therefore, the above memory management method is also called a static memory management method.

[0069] (5) Swap.

[0070] The above static memory management method avoids the extra overhead caused by memory reallocation and release, and can also avoid the problem of memory fragmentation. However, even if memory reuse is applied, the memory wall problem is still severe for large AI models, and other memory saving technologies need to be combined to further compress the memory space occupied by tensors.

[0071] Swap is a technology that can effectively compress the memory space occupied by tensors. This technology is mainly based on the hardware architecture of the neural network computing system. It swaps the intermediate results generated by the computing operations of the AI ​​large model to the memory outside the neural network chip, for example, swapping the intermediate results to the host memory, thereby reducing the memory occupied by the neural network chip.

[0072] like Figure 5 As shown, the hardware architecture of the neural network computing system includes a neural network chip and a host. The neural network chip and the host have a communication connection. Tensors can be transferred from the neural network chip memory to the host memory through the communication connection, and can also be transferred from the host memory to the neural network chip memory. The computational graph of the neural network includes operator 1-FP, operator 1-BP, operator 2-FP and operator 2-BP. Operator 1-FP is operator 1 in the forward propagation calculation process, operator 1-BP is operator 1 in the reverse propagation calculation process, operator 2-FP is operator 2 in the forward propagation calculation process, and operator 2-BP is operator 2 in the reverse propagation calculation process. ΔY1 and ΔY2 are the results generated in the reverse propagation calculation.

[0073] Operator 1-FP needs to use tensor x. Before operator 1-FP is executed, tensor x is transferred from the host memory to the neural network chip memory. After operator 1-FP is executed, tensor x is transferred from the host memory to the neural network chip memory.

[0074] Tensor y is the result generated by operator 1-FP. Both operator 2-FP and operator 2-BP need to use tensor y. Tensor y can be transferred from the neural network chip memory to the host memory after operator 2-FP is executed, and before operator 2-BP is executed, tensor y can be transferred from the host memory to the neural network chip memory.

[0075] The above-mentioned operation of transferring from the neural network chip memory to the host memory can be called swap out, and the above-mentioned operation of transferring from the host memory to the neural network chip memory can be called swap in.

[0076] Figure 5 In the memory management method shown, memory allocation occurs in the calculation stage of the neural network, so the memory management method is also called a dynamic memory management method. Before the dynamic memory management method is implemented, a portion of the memory space of the neural network chip is first divided as a memory pool. When memory space needs to be allocated to a tensor, it is first allocated from the memory pool. If the allocation is successful, the allocated memory space will be returned to the memory pool after the life cycle of the tensor ends; if the allocation fails, it means that the remaining memory space in the memory pool is insufficient, and another memory space can be applied from the remaining memory space of the neural network chip memory and added to the memory pool until all the memory of the neural network chip is allocated.

[0077] Figure 6 The change of memory usage after applying the dynamic memory management method is shown.

[0078] Currently, there are four tensors, namely tensor m, tensor n, tensor x and tensor y. Among them, tensor m and tensor n are tensors that do not need to perform swap operations, while tensor x and tensor y are tensors that need to perform swap operations. If the swap operation is not performed, these four tensors will occupy the neural network chip memory during their respective life cycles; if the swap operation is performed, tensor x and tensor y can be swapped from the neural network chip memory to the host memory. For the neural network chip memory, the memory space occupied by the swapped tensors can be released, and the released memory space can be added to the memory pool, thereby saving the neural network chip memory.

[0079] Dynamic memory management methods include dynamic allocation and release of memory. The allocation and release operations will bring additional memory management overhead, and the allocation and release operations may block other operations (such as neural network computing operations), resulting in a decrease in the training / inference speed of the neural network chip. In addition, dynamic memory management will also cause memory fragmentation problems, that is, when the total amount of free memory is sufficient for tensors to use, the free memory space is not continuous, resulting in the problem of being unable to allocate free memory to tensors.

[0080] (6) Neural network computing system.

[0081] Figure 7 A neural network computing system suitable for an embodiment of the present application is shown, which may include a neural network chip 710 and a host 720. The neural network chip 710 and the host 720 may be connected via a communication interface 730, such as an extended peripheral component interconnect express (PCIe) interface.

[0082] The host 720 may include a processor 721 and a memory 722 .

[0083] Processor 721 is the computing core and control core of host 720, and can access memory 722 and peripheral devices (such as neural network chip 710). Processor 721 can be a central processing processor (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP) or other programmable logic devices, such as discrete gates, transistor logic devices or discrete hardware components.

[0084] The memory 722 is usually used to store computing data and control instructions. The memory 722 can be connected to the processor 721 through a double data rate (DDR) bus. In order to improve the access speed of the processor 721, the memory 722 needs to have a fast access speed. The processor 721 can access the memory through a memory controller ( Figure 7 The memory 722 is accessed by the user (not shown) to perform a read operation or a write operation on a storage unit in the memory 722.

[0085] The memory 722 may be a volatile memory or a nonvolatile memory, or the memory 722 may include both volatile memory and nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. As an optional example, the RAM can be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) or direct memory bus random access memory (direct rambus RAM, DRRAM).

[0086] The host 720 may also include devices such as a disk, and the embodiment of the present application does not limit the specific architecture of the host 720.

[0087] The neural network chip 710 may include a processor 713 and a memory 711 .

[0088] Processor 713 is used to run operators to process data related to the neural network. Processor 713 can be a streaming multiprocessor, which can include at least one execution unit (execution unit), such as a compute unified device architecture (CUDA) unit.

[0089] As an optional example, the processor 713 may be a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processing unit (NPU).

[0090] The memory 711 is used to store data and instructions related to the neural network calculation process. The memory 711 can be a volatile memory, and the volatile memory can be a RAM used as a cache.

[0091] As an optional example, the memory 711 may be a high bandwidth memory (HBM) or a graphics double data rate (GDDR) memory.

[0092] As an optional example, the processor 713 and the memory 711 can be a storage and computing integrated device, such as a memristor, where the memristor includes but is not limited to the following types: resistive random access memory (ReRAM), conductive-bridging RAM (CBRAM) or phase-change memory (PCM).

[0093] The embodiment of the present application does not limit the specific types of the processor 713 and the memory 711.

[0094] In some cases, neural network chip 710 also includes processor 714 and memory 712.

[0095] The processor 714 is a processor with a processing capability weaker than that of the processor 721, and is used to provide part of the functions of the host 720. The processor 714 may be a central processing processor (CPU) or other logic devices.

[0096] The memory 712 is generally used to store calculation data and control instructions of the processor 714. The memory 712 may be a volatile memory or a non-volatile memory, or the memory 712 may include both a volatile memory and a non-volatile memory.

[0097] The processor 714 and the memory 712 may be two independent devices or an integrally packaged device. The embodiment of the present application does not limit the specific types of the processor 714 and the memory 712.

[0098] The various devices inside the neural network chip 710 can communicate through the communication interface 740, and the communication interface 740 can be a DDR bus.

[0099] The neural network chip 710 may also include devices such as a memory controller. The embodiment of the present application does not limit the specific architecture of the neural network chip 710.

[0100] In the above system, the memory used to store data and instructions related to the neural network calculation process (such as memory 711) can be called neural network chip memory, and other memories (such as memory 712 and memory 722) are called non-neural network chip memory.

[0101] (7)AI big model.

[0102] AI big models are neural networks with huge parameters and usually have the following characteristics:

[0103] 1. A large number of parameters.

[0104] The parameter scale of AI big models has exceeded 100 billion and is developing towards 1 trillion scale. These parameters can store the knowledge and experience of AI big models. More parameters mean that AI big models have stronger learning and representation capabilities, and can better capture the characteristics of data and the relationship between data for reasoning and prediction.

[0105] 2. Context understanding and generation.

[0106] AI big models are able to understand and generate content with context and semantics. They learn and train large amounts of input data such as language and images through key technologies such as attention recognition mechanisms, and can extract useful information from complex real-world scenarios.

[0107] 3. Powerful generalization ability.

[0108] AI big models have strong generalization capabilities through training on large-scale data. They can learn a wide range of features and patterns from a large amount of data, and can apply the learned features and patterns to the processing of data they have never encountered.

[0109] 4. Transfer learning capability.

[0110] AI big models are pre-trained in one or more fields and can transfer the learned knowledge to new tasks or new fields. This migration capability makes AI big models learn faster on new tasks and also improves the performance of AI big models in unknown fields.

[0111] 5. Application in multiple fields.

[0112] AI big models have a wide range of applications and can be applied to multiple fields and solve multiple tasks, such as natural language processing, computer vision, and speech recognition. AI big models not only have strong capabilities in a single modality, but can also handle tasks across multiple modalities.

[0113] While AI big models have many advantages, they also have some challenges and limitations, such as long training time, large computing resource requirements, and high model complexity. Among them, AI big models require powerful computing resources for parameter optimization and reasoning, and the demand for computing resources is particularly urgent, which poses challenges to both hardware and software capabilities.

[0114] Figure 8 It is an AI large model software protocol stack provided in an embodiment of the present application.

[0115] From top to bottom, the AI ​​big model software protocol stack includes the algorithm layer, front-end compilation layer, back-end compilation layer, runtime and operator layer. Figure 8 This is an example of an AI large model software protocol stack. The AI ​​large model software protocol stack may also include more layers or fewer layers, which is not limited to the embodiments of the present application.

[0116] The algorithm layer is used to receive large AI models, which are usually input into the algorithm layer in the form of computational graphs.

[0117] The front-end compilation layer, back-end compilation layer, and runtime are mainly used to optimize the operating resources of large AI models. For example, the front-end compilation layer can determine the recomputation strategy and swap strategy based on the calculation graph of the large AI model. The back-end compilation layer can determine the memory reuse technology based on the recomputation strategy and swap strategy. The runtime can provide high-performance data exchange technology and perform data exchange based on memory reuse technology.

[0118] The operator layer includes the operator library, and its main function is to call operators from the operator library to perform computing operations based on the computational graph of the AI ​​large model.

[0119] The above article introduces two memory management methods used in neural network calculations, namely, memory reuse and swap. Usually, the neural network computing system divides the memory into two parts, one for memory reuse and the other for swap. For memory reuse, since different tensors occupy different memory spaces, some memory space cannot be reused; for swap, there is also the problem of memory space waste. For example, after tensor x is swapped out, the remaining memory space is too large for other tensors, resulting in part of the memory space being wasted.

[0120] The following describes a method for reusing neural network chip memory provided in an embodiment of the present application.

[0121] like Fig. 9 As shown, the execution subject of method 900 may be a neural network computing device, such as host 720, or a processor applied to the neural network computing device, such as processor 721. The following description is made by taking the execution subject as a neural network computing device (referred to as "computing device") as an example. Method 900 includes:

[0122] S910, determine the target tensor, where the target tensor is the tensor that needs to be exchanged from the neural network chip memory to the non-neural network chip memory.

[0123] The calculation process of a neural network involves a variety of tensors. Some tensors are frequently used in the calculation process. These tensors reside in the memory of the neural network chip and do not need to be exchanged. Other tensors are intermediate data generated during the calculation process of the neural network. These tensors will no longer be used in the short term and can be exchanged. The tensors that need to be exchanged from the neural network chip memory to the non-neural network chip memory are the target tensors.

[0124] The target tensor may be one tensor or multiple tensors. The embodiment of the present application does not limit the specific type of the target tensor and the method for determining the target tensor.

[0125] Optionally, the target tensor is a tensor that cannot be generated multiple times during the computation of the neural network.

[0126] Some tensors can be generated multiple times through computing operations. When such tensors need to be used, they can be directly generated through computing operations without performing exchange operations on such tensors, thus saving the exchange overhead.

[0127] For example, mixed-precision computing is a common method in neural network training. It uses a mixture of single-precision data types and half-precision data types to convert some computing operations from single-precision to half-precision to speed up the calculation. Among the identified tensors that occupy dynamic memory, there is a type of half-precision tensor converted from single-precision parameter tensors stored in static memory in a mixed-precision scenario. Its life cycle starts from the forward calculation and ends after the reverse calculation is completed. This type of tensor is often large in size and brings high overhead to swap. This type of parameter tensor can be regenerated from static memory through a data type conversion (Cast) operation instead of a swap operation. This processing method is called recalculation.

[0128] like Fig.10 As shown, F represents the neural network layer involved in the forward calculation, and G represents the neural network layer involved in the reverse calculation. For example, F (i) represents the i-th neural network layer involved in the forward calculation, G (i) Indicates the i-th neural network layer involved in the reverse calculation. W represents weight, G represents gradient, PF32 represents 32-bit floating point number, and PF16 represents 16-bit floating point number. W FP32 and G FP32 are full-precision tensors, and W FP16 and G FP16 are half-precision tensors. OPT represents the optimizer parameter update operation, and L represents the output value of the loss function. The swap-marker operator is used to record the metadata of tensors that need to be swapped during the forward calculation process, and the swap-marker-grad operator is used to record the metadata of tensors that need to be swapped during the reverse calculation process. The functions of these two operators will be introduced below.

[0129] When performing forward calculations in the i-th layer of the neural network, the full-precision tensor W FP32 is converted to the half-precision tensor W FP16 input F through the Cast operation. (i) After the neural network completes the forward calculation, L is obtained. Subsequently, reverse calculation is performed based on L. After the i-th layer of the neural network performs reverse calculation, it outputs a half-precision tensor G FP16. G FP16 is converted to a full-precision tensor G FP32 through a Cast operation for OPT operation. The result of the OPT operation is used to update W FP32.

[0130] W FP16 is also used when performing reverse calculations on the i-th layer of the neural network, so when recalculation is not applied, F (i) After the calculation is completed, W FP16 is stored in the neural network chip memory or host memory until G (i)It is discarded after the calculation is completed (that is, deleted from the neural network chip memory or host memory). In other words, the life cycle of W FP16 starts from F (i) The calculation starts at the time G (i) The calculation end time is Fig.10 Shown in the upper part.

[0131] W FP32 is a tensor that exists throughout the training process. It is stored in static memory and is read from F (i) To G (i) During the process, the W FP32 tensor is not modified, so in G (i) When W FP16 is needed, you can use W FP32 to generate W FP16 instead of always saving W FP16.

[0132] The training process after applying recalculation is as follows Fig.10 As shown in the lower part of the figure, when the neural network layer i is forward-computing, the full-precision tensor W FP32 is converted to the half-precision tensor W FP16 input F through the Cast operation. (i) . F (i) After the calculation is completed, W FP16 is discarded and G (i) Before the calculation begins, the neural network chip can perform a Cast operation (i.e., apply recalculation) to convert W FP32 to W FP16 input G (i) It can be seen that W FP16 can be generated when needed and discarded after use, without the need to perform a swap operation on W FP16.

[0133] When determining the target tensor, tensors such as W FP16 that can be generated multiple times can be excluded to save swap overhead.

[0134] S920, determining the life cycle of the target tensor based on multiple computing operations in which the target tensor participates, wherein the life cycle of the target tensor is greater than or equal to the sum of the durations of the multiple computing operations, and the life cycle of the target tensor is less than the length of time from the start time of the multiple computing operations to the end time of the multiple computing operations.

[0135] by Fig.11 For example, the target tensor participates in the forward and reverse calculations of the i-th layer of the neural network. The target tensor is in F (i) After the calculation of is completed, it will no longer participate in the neural network calculation for a long time and can be swapped to the non-neural network chip memory. If the life cycle of the target tensor is not processed, the life cycle of the target tensor is equal to the length of its existence, that is, from F (i) The calculation starts at the time G (i) The calculation end time is Fig.11 shown in the upper part of .

[0136] In F (i) The calculation ends at G (i) During the period from the start of the calculation to the time when the target tensor is not in the memory of the neural network chip, so F (i) The calculation ends at G (i) It is inappropriate to include the calculation start time in the life cycle of the target tensor, and the life cycle of the target tensor needs to be redefined.

[0137] The time that the target tensor participates in the computation of the neural network can be defined as the life cycle of the tensor, such as Fig.11 As shown in the lower part of , the lifetime of the target tensor is equal to F (i) and G (i) In this way, the life cycle of the target tensor is shortened. When memory reuse is applied, a smaller memory space can be reused, thereby improving the memory usage efficiency of the neural network chip.

[0138] Optionally, the life cycle of the target tensor may also be greater than the time during which the target tensor participates in the computational operation of the neural network.

[0139] like Fig.12 As shown, the target tensor participates in the forward and reverse calculations of the i-th layer of the neural network, as well as the forward and reverse calculations of the i+1-th layer of the neural network. The target tensor is in F (i) After the calculation of F is completed, (i+1) Therefore, the target tensor is in F (i) After the calculation is completed, no swap out operation is performed, but in F (i+1) After the calculation of G is completed, the swap out operation is performed. Similarly, the target tensor is in G (i+1) Before the calculation of G starts, swap in operation is performed from non-neural network chip memory to neural network chip memory. (i+1) With G (i) The calculation interval between G is very short. (i+1) After the calculation is completed, the target tensor is no longer swapped out.

[0140] Fig.12 In the example above, the life cycle of the target tensor includes F (i) With F (i+1) The calculation interval between G (i) With G (i+1) Therefore, the lifetime of the target tensor is greater than F (i) 、F (i+1) , G (i) and G(i+1) However, the life cycle of the target tensor is still shorter than that without redefinition. When memory reuse is applied, a smaller memory space can still be reused to improve the memory usage efficiency of the neural network chip.

[0141] S930, reuse the neural network chip memory according to the life cycle of the target tensor.

[0142] like Fig.13 As shown, there are currently four tensors T1, T2, T3 and T4. T1 and T3 are the tensors to be swapped, that is, the tensors that need to be swapped out and swapped in, and T2 and T4 are tensors that do not need to be swapped. It can be determined that T1 and T3 are the target tensors.

[0143] If the life cycles of T1 and T3 are not redefined, the life cycles of T1 and T3 are longer, the life cycle of T1 overlaps with the life cycles of T2 and T4 respectively, and the life cycle of T3 also overlaps with the life cycles of T2 and T4 respectively. It is difficult for T2 and T4 to reuse the neural network chip memory of T1 and T3, resulting in a waste of memory space.

[0144] according to Fig.11 or Fig.12 After the method shown redefines the life cycle of T1 and T3, the life cycle of T1 and T3 is shortened, and T1 and T3 can reuse the memory space of T2 and T4. Fig.13 As shown on the right side of , T1 is redefined as T 11 and T 12 (actually T1 at a different time), T 11 and T 12 The life cycle of T3 no longer overlaps with that of T2, so T2 can reuse the neural network chip memory of T1. 31 and T 32 (actually T3 at a different time), T 31 and T 32 The life cycle of T3 no longer overlaps with that of T4, so T4 can reuse the neural network chip memory of T3.

[0145] Fig.13 The memory reuse method shown is an example rather than a limitation, and the embodiments of the present application do not limit the specific method of memory reuse.

[0146] To summarize, in this embodiment, the life cycle of the target tensor is shorter than the duration from the start time of multiple computing operations to the end time of multiple computing operations. In the interval between these multiple computing operations, the target tensor does not participate in the computing operations. The computing device exchanges the target tensor from the neural network chip memory to the non-neural network chip memory, and by redefining the life cycle of the target tensor, the idle neural network chip memory is allocated to the tensors participating in the computing operations, thereby improving the utilization efficiency of the neural network chip memory.

[0147] In the method described above, there is no restriction on the swap operation. The computing device can perform a swap operation on each tensor to be swapped. However, the swap operation will generate data movement overhead. If a swap operation is performed on each tensor to be swapped, a large amount of data movement overhead will be generated. The following describes a method for reducing data movement overhead.

[0148] Optionally, the method 900 further includes:

[0149] Determine a plurality of tensors, each of which is a tensor that needs to be exchanged from a neural network chip memory to a non-neural network chip memory, and the plurality of tensors includes a target tensor;

[0150] Accordingly, S930 includes:

[0151] Based on the life cycles of multiple tensors, multiple tensors are exchanged from the neural network chip memory to the non-neural network chip memory through a single exchange operation; and / or, based on the life cycles of multiple tensors, multiple tensors are exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

[0152] like Fig.14 As shown, each computing operation related to the neural network forms a computing flow. The swap-marker operator is used to record the metadata of the tensor that needs to be swapped during the forward calculation process, and the swap-marker-grad operator is used to record the metadata of the tensor that needs to be swapped during the reverse calculation process. The metadata records the characteristics of the tensor. For example, the metadata may include layer index (layer_id), tensor index (tensor_id), tensor size (size), host address (host-addr), and neural network chip address (dev-addr), etc. The embodiments of the present application do not limit the specific form of the metadata.

[0153] During the neural network calculation process, the computing device will sequentially scan the computing operations on the computing flow. When the swap-marker operator is scanned, the metadata recorded by it will be pushed to the swap out queue. When the amount of metadata in the swap out queue exceeds the quantity threshold, the computing device can swap the tensor corresponding to the metadata in the swap out queue to the non-neural network chip memory. Alternatively, when the computing operation of a neural network layer is completed, the computing device swaps the tensor corresponding to the metadata in the swap out queue with the same layer index as the neural network layer to the non-neural network chip memory.

[0154] For example, swap-marker operator i records F (i) The metadata of the tensor to be used, after the computing device scans the swap-marker operator i, pushes its recorded metadata to the swap out queue. (i) After the calculation operation is completed, the computing device swaps out the queue with F (i) The tensors corresponding to the same metadata of the layer index are exchanged to the non-neural network chip memory to facilitate the reuse of the neural network chip memory of these tensors.

[0155] Optionally, you can swap out the queue with F (i) The tensor with the same metadata as the layer index is called the first tensor. The first tensor is in F (i) is determined before the calculation operation is completed, and, at F (i+1) The computing device can perform the swap out operation through two threads respectively. (i+1) The calculation operation and the swap out operation of the first tensor, so that F (i+1) The computation of the tensor does not need to wait for the swap out operation of the first tensor, thus reducing the negative impact of the swap operation on the training speed.

[0156] Similarly, when the computing device scans the swap-marker-grad operator, it pushes its recorded metadata to the swap in queue. When the amount of metadata in the swap in queue exceeds the quantity threshold, the computing device can swap the tensors corresponding to the metadata in the swap in queue to the non-neural network chip memory. Alternatively, before the computing operation of a neural network layer begins, the computing device swaps the tensors corresponding to the metadata in the swap in queue with the same layer index as the neural network layer to the neural network chip memory.

[0157] For example, the swap-marker-grad operator i+1 records G (i)The metadata of the tensor to be used, after the computing device scans the swap-marker-grad operator i+1, pushes its recorded metadata to the swap in queue. (i) Before the calculation operation starts, the computing device swaps the G (i) The tensors corresponding to the same metadata of the layer index are exchanged to the neural network chip memory to facilitate G (i) Complete the calculation.

[0158] The computing device may implement “performing a swap out operation on multiple tensors to be exchanged”; or, the computing device may implement “performing a swap in operation on multiple tensors to be exchanged”; or, the computing device may implement “performing a swap out operation on multiple tensors to be exchanged” and “performing a swap in operation on multiple tensors to be exchanged”.

[0159] Optionally, you can swap in the queue with G (i) The tensor with the same metadata as the layer index is called the first tensor. The first tensor in G (i+1) is determined before the calculation operation is completed, and, in G (i+1) The computing device can perform G through two threads respectively. (i+1) The calculation operation and the swapout operation of the first tensor, so that G (i+1) The computation of the tensor does not need to wait for the swap out operation of the first tensor, thus reducing the negative impact of the swap operation on the training speed.

[0160] In addition to dedicated processors for tensor calculations, some neural network chips also include one or more general-purpose processors with weaker processing capabilities. The general-purpose processors are also configured with corresponding memory space, which can also be used as non-neural network chip memory.

[0161] Below Figure 7 The system shown in FIG. 7 is used as an example for explanation. Processor 713 is a dedicated processor for tensor calculation, and memory 711 is a memory configured for processor 713. Processor 714 is a general-purpose processor whose processing power is weaker than that of processor 721, and memory 712 is a memory configured for processor 714. Memory 711 can be called a neural network chip memory, and memory 712 and memory 722 can be called non-neural network chip memory.

[0162] Exemplarily, processor 714 is an AICPU, memory 712 is an AICPU DDR memory (referred to as AICPU DDR for short), memory 722 is a host DDR memory (referred to as host DDR for short), memory 711 is HBM, the communication interface 730 between HBM and host DDR is a PCIe interface, and the communication interface 740 between HBM and AICPU DDR is a DDR bus.

[0163] The transfer rate of the PCIe interface is usually lower than that of the DDR bus. For example, the transfer rate of the PCIe-3 interface can reach 16 GB / s, while the transfer rate of the DDR bus can reach 200 GB / s. This hierarchical memory structure brings opportunities for memory saving.

[0164] Optionally, when exchanging multiple tensors from the neural network chip memory to the non-neural network chip memory, the computing device may perform the following steps:

[0165] Exchange the tensor with a shorter calculation interval among the multiple tensors to the first memory;

[0166] A tensor with a longer calculation interval among multiple tensors is exchanged to the second memory; wherein the calculation interval is the interval time between the first time any tensor among the multiple tensors participates in a calculation operation and the last time it participates in a calculation operation.

[0167] The longer the calculation interval of the tensor, the longer the time available for the exchange operation, and a lower transmission rate can also be used to exchange the tensor to the neural network chip memory before the calculation operation is performed. The shorter the calculation interval of the tensor, the shorter the time available for the exchange operation, and a higher transmission rate is required to exchange the tensor to the neural network chip memory before the calculation operation is performed. In this embodiment, tensors with longer calculation intervals are exchanged to memory with lower transmission rates, and tensors with shorter calculation intervals are exchanged to memory with higher transmission rates. This can reasonably utilize non-neural network chip memory to meet the exchange requirements of different tensors.

[0168] like Fig.15 As shown, F represents the neural network layer involved in the forward calculation, and G represents the neural network layer involved in the reverse calculation. For example, F (1) Indicates the first neural network layer involved in the forward calculation, F (L) represents the Lth neural network layer involved in the forward calculation (also the last neural network layer involved in the forward calculation); G (L) Indicates the first neural network layer involved in the reverse calculation, G (1) represents the Lth neural network layer involved in the reverse calculation (also the last neural network layer involved in the reverse calculation). Wherein, L and l are both positive integers, L is greater than l, and Ll is greater than 1.

[0169] Tensor 1 is the number of participants in F (1) and G (1) The tensor of the calculation, tensor Ll-1 is the tensor involved in F (L-l-1) and G (L-l-1) The tensor Ll is the tensor involved in F (L-l) and G (L-l) The tensor of the calculation, tensor L is the tensor involved in F (L) and G (L) The tensor of the computation. Fig.15 As can be seen, tensor 1 participates in F (1) After the calculation, it takes a long time to participate in G (1) Tensor 1 has enough time to perform swap out and swap in operations; Tensor L participates in F (L) After the calculation, it is necessary to participate in G (L) The time for swapping out and swapping in of tensor L is very short.

[0170] It can be seen that the calculation intervals of different neural network layers are different. This feature can be used to determine whether the tensors of the neural network layer need to be exchanged to the AICPU DDR or the host DDR. Tensors with longer calculation intervals (such as tensor L) can be exchanged from HBM to the host DDR with a slower transmission rate, and tensors with shorter calculation intervals (such as tensor L) can be exchanged from HBM to the AICPU DDR with a faster transmission rate. In this way, non-neural network chip memory can be reasonably used to meet the exchange requirements of different tensors.

[0171] Optionally, the tensor with shorter calculation interval is the tensor corresponding to the neural network layer after the target neural network layer, and the tensor with longer calculation interval is the tensor corresponding to the target neural network layer and the neural network layer before the target neural network layer.

[0172] The number of the target neural network layer can be determined based on experience. Fig.15 In the example shown, the target neural network layer is F (L-l-1) . The later the neural network layer is positioned, the smaller the calculation interval between the forward calculation operation and the reverse calculation operation, and the tensors to be exchanged corresponding to these neural network layers need to use memory with a higher transmission rate. The earlier the neural network layer is positioned, the larger the calculation interval between the forward calculation operation and the reverse calculation operation, and the tensors to be exchanged corresponding to these neural network layers can use memory with a lower transmission rate. Therefore, this embodiment can reasonably utilize non-neural network chip memory to meet the exchange requirements of different tensors.

[0173] Optionally, the ordinal number of the target neural network layer can be determined by the following formula:

[0174]

[0175] In the above formula, s represents the number of neural network layers after the target neural network layer, M AICPU represents the size of AICPUDDR, D represents the size of the tensor to be exchanged in a single neural network, and L represents the number of layers of the neural network.

[0176] like Fig.15 As shown, s is equal to Ll, indicating that the number of neural network layers after the target neural network layer is Ll, ​​so it can be determined that the target neural network layer is Ll-1.

[0177] The larger it is, the larger the capacity of AICPU DDR is. For example, If s is 3, it means that the AICPU DDR can carry the tensors to be exchanged of three neural network layers. If the number of neural network layers is 2, then s is equal to 2, indicating that the tensors to be exchanged of all neural network layers can be exchanged to the first memory; if the number of neural network layers is 6, then s is equal to 3, indicating that the tensors to be exchanged of some neural network layers (the last 3 neural network layers) can be exchanged to the AICPU DDR. Therefore, this embodiment can make full use of the high transmission rate of the AICPU DDR, while avoiding exchange errors caused by excessive data volume of the tensors to be exchanged.

[0178] The following is an example of another method for reusing neural network chip memory provided by the present application.

[0179] like Fig.16 As shown, the method 1600 may be executed by the processor 721 or the processor 714, and the following description is made taking the processor 721 as an example. The method 1600 includes:

[0180] S1610, input the computation graph.

[0181] An example of a computational graph is Figure 3 As shown, after the processor 721 obtains the calculation graph, it can perform the following steps.

[0182] S1620, determine the tensors in the computation graph that need to perform swap operations, and perform redefine operations on these tensors.

[0183] The processor 721 can determine the tensors that need to perform the swap operation according to the user-customized swap strategy, and mark these tensors through the swap-marker operator and the swap-marker-grad operator. The embodiment of the present application does not limit the specific method of determining the need to perform the swap operation.

[0184] The processor 721 may Fig.11 or Fig.12 The method shown performs redefinition operations on tensors that need to perform swap operations, shortening the life cycle of these tensors.

[0185] S1630: Among the tensors that need to perform the swap operation, are there any tensors that can perform the recalculation operation?

[0186] The processor 721 may Fig.10 The method shown determines whether the tensor can be recalculated. If so, S1640 can be executed. If not, S1650 can be executed.

[0187] S1640, update the tensors in the computation graph that need to perform swap operations.

[0188] For example, after executing S1620, the processor 721 determines N tensors that need to perform swap operations, and after executing S1630, determines M tensors that need to perform swap operations, then the processor 721 can remove the swap marks of the M tensors and mark them with recalculation marks, thereby updating the tensors that need to perform swap operations in the calculation graph. Wherein, M and N are both positive integers, and M is greater than N.

[0189] S1650: Determine a memory reuse strategy.

[0190] Memory reuse strategy such as Figure 4 or Fig.13 As shown, the processor 721 may also determine the memory reuse strategy through other heuristic methods, and the embodiments of the present application do not limit the specific content of the memory reuse strategy.

[0191] S1660, executes asynchronous batch swap and memory hierarchical swap.

[0192] An example of asynchronous batch swap is as follows Fig.14 As shown, an example of memory hierarchical swap is as follows Fig.15 As shown, no further description is given here.

[0193] In method 1600, redefinition shortens the life cycle of tensors and allows more memory to be reused; recalculation reduces the number of tensors that need to perform swap operations and allows more memory to be reused; asynchronous batch swap allows multiple tensors that need to perform swap operations (such as tensors that need to perform swap operations in a neural network layer) to be packaged for data transfer, thereby improving data transfer efficiency and avoiding the blocking of computing operations by swap operations; memory hierarchical swap can reasonably utilize non-neural network chip memory with different transmission rates to meet the exchange requirements of different tensors.

[0194] Fig.17This is a diagram showing the effect of applying the method of reusing the neural network chip memory according to an embodiment of the present application.

[0195] Fig.17 The figure shows the dynamic memory size occupied by different stages in the pipeline dimension in the process of training AI large models using 3D parallel technology after applying three memory management methods respectively. The horizontal axis represents the different stages of pipeline parallelism, and the vertical axis represents the dynamic memory usage.

[0196] Depend on Fig.17 It can be seen that when the swap technology is not applied, the memory usage is the largest no matter which stage the AI ​​large model training is in. After applying the swap technology, the memory usage has been significantly reduced compared to the case where the swap technology is not applied. The average memory usage saved in multiple stages is about 2GB. At this time, the amount of data of the tensor performing the swap operation is about 10GB, indicating that the decrease in memory usage is limited. This is mainly because the tensors that perform the swap operation are managed separately from the tensors that do not perform the swap operation, resulting in low memory reuse efficiency. In addition, since asynchronous batch swap and memory hierarchical swap are not applied, the swap overhead brought about by the application of the swap technology causes the training speed to be about 11% slower than the training speed when the swap technology is not applied.

[0197] After applying the method of reusing the memory of the neural network chip provided by the present application, the memory usage is further reduced compared to the case of applying the swap technology. Compared with the case of applying the swap technology, the average amount of memory usage saved in multiple stages is about 4GB. In addition, thanks to asynchronous batch swap and memory hierarchical swap, swap overhead no longer affects the training speed.

[0198] The above describes in detail an example of a method for reusing the memory of a neural network chip provided by the present application. It is understandable that, in order to implement the above functions, the corresponding device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0199] The present application can divide the functional units of the device for reusing the neural network chip memory according to the above method example. For example, each function can be divided into each functional unit, or two or more functions can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in this application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0200] Fig.18 1800 is a schematic diagram of a device for reusing the memory of a neural network chip provided by the present application. The device 1800 includes a processing unit 1810.

[0201] Processing unit 1810 is used to: determine a target tensor, which is a tensor that needs to be exchanged from a neural network chip memory to a non-neural network chip memory; determine the life cycle of the target tensor based on multiple computing operations in which the target tensor participates, wherein the life cycle of the target tensor is greater than or equal to the sum of the durations of the multiple computing operations, and the life cycle of the target tensor is less than the length of time from the start time of the multiple computing operations to the end time of the multiple computing operations; and reuse the neural network chip memory based on the life cycle of the target tensor.

[0202] Optionally, the target tensor is a tensor that cannot be generated multiple times during the calculation process of the neural network.

[0203] Optionally, processing unit 1810 is also used to: determine multiple tensors, each of which is a tensor that needs to be exchanged from the neural network chip memory to the non-neural network chip memory, and the multiple tensors include the target tensor; processing unit 1810 is specifically used to: exchange the multiple tensors from the neural network chip memory to the non-neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors; and / or exchange the multiple tensors from the non-neural network chip memory to the neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors.

[0204] Optionally, processing unit 1810 is specifically used to: exchange the multiple tensors from the neural network chip memory to the non-neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors; the multiple tensors include a first tensor participating in a forward computing operation of a first neural network layer; determine the first tensor before the forward computing operation of the first neural network layer is completed; after the forward computing operation of the first neural network layer is completed and before the first tensor is exchanged, processing unit 1810 is also used to: execute a forward computing operation of a second neural network layer, the second neural network layer being a neural network layer located after the first neural network layer.

[0205] Optionally, processing unit 1810 is specifically used to: exchange the multiple tensors from the non-neural network chip memory to the neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors; the multiple tensors include a first tensor participating in a reverse computing operation of a first neural network layer; determine the first tensor before the reverse computing operation of a second neural network layer is completed, and the second neural network layer is a neural network layer located after the first neural network layer; before the reverse computing operation of the second neural network layer is completed, exchange the first tensor from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

[0206] Optionally, the non-neural network chip memory includes a first memory and a second memory, and the transmission rate of the first memory is higher than the transmission rate of the second memory; the processing unit 1810 is specifically used to: exchange the tensors with shorter calculation intervals among the multiple tensors to the first memory; exchange the tensors with longer calculation intervals among the multiple tensors to the second memory; wherein the calculation interval is the interval time between the first time any tensor among the multiple tensors participates in a calculation operation and the last time it participates in a calculation operation.

[0207] Optionally, the tensor with a shorter calculation interval is a tensor corresponding to the neural network layer after the target neural network layer, and the tensor with a longer calculation interval is a tensor corresponding to the target neural network layer and the neural network layer before the target neural network layer.

[0208] Optionally, the number of neural network layers after the target neural network layer is equal to the smaller of a first value and the number of neural network layers, and the first value is the result of rounding down the ratio of the capacity of the first memory to the amount of data of the tensor to be exchanged corresponding to the single-layer neural network.

[0209] Those skilled in the art can clearly understand that the specific working process of the device 1800 and the technical effects produced by the execution steps can be referred to the description in the corresponding method embodiment above, and for the sake of brevity, they will not be repeated here.

[0210] The device 1800 may be an electronic device or a chip. The processing unit 1810 may be implemented by hardware or software. When implemented by hardware, the processing unit 1810 may be a logic circuit, an integrated circuit, etc.; when implemented by software, the processing unit 1810 may be a general-purpose processor implemented by reading software codes stored in a storage unit. The storage unit may be integrated in the processing unit 1810 or located outside the processing unit 1810 and exist independently.

[0211] Fig.19 A structural schematic diagram of an electronic device provided by the present application is shown. Fig.19 The dotted line in the figure indicates that the unit or the module is optional. The device 1900 can be used to implement the method described in the above method embodiment. The device 1900 can be a terminal device, a server, or a chip.

[0212] The device 1900 includes one or more processors 1901, which can support the device 1900 to implement the method in the method embodiment. The processor 1901 can be a general-purpose processor or a special-purpose processor. For example, the processor 1901 can be a central processing unit (CPU). The CPU can be used to control the device 1900, execute software programs, and process data of software programs. The device 1900 can also include a communication unit 1905 to implement signal input (reception) and output (transmission).

[0213] For example, device 1900 may be a chip, communication unit 1905 may be an input and / or output circuit of the chip, or communication unit 1905 may be a communication interface of the chip, and the chip may be a component of a terminal device, a server, or other electronic device.

[0214] For another example, the device 1900 may be a terminal device or a server, and the communication unit 1905 may be a transceiver of the terminal device or the server, or the communication unit 1905 may be a transceiver circuit of the terminal device or the server.

[0215] The device 1900 may include one or more memories 1902, on which a program 1904 is stored. The program 1904 can be executed by the processor 1901 to generate instructions 1903, so that the processor 1901 performs the method described in the above method embodiment according to the instructions 1903. Optionally, data may also be stored in the memory 1902. Optionally, the processor 1901 may also read the data stored in the memory 1902, and the data may be stored at the same storage address as the program 1904, or the data may be stored at a different storage address from the program 1904.

[0216] The processor 1901 and the memory 1902 may be provided separately or integrated together, for example, integrated on a system on chip (SOC) of a terminal device.

[0217] The specific manner in which processor 1901 executes the method of reusing the neural network chip memory can be found in the relevant description in the method embodiment.

[0218] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in the processor 1901. The processor 1901 can be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0219] The present application also provides a computer program product, which, when executed by the processor 1901, implements the method described in any method embodiment of the present application.

[0220] The computer program product may be stored in the memory 1902 , for example, a program 1904 , which is converted into an executable target file that can be executed by the processor 1901 after preprocessing, compilation, assembly, and linking.

[0221] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method described in any method embodiment of the present application is implemented. The computer program can be a high-level language program or an executable target program.

[0222] The computer-readable storage medium is, for example, memory 1902. Memory 1902 may be a volatile memory or a nonvolatile memory, or memory 1902 may include both a volatile memory and a nonvolatile memory. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0223] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless or microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0224] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the electronic device can perform some or all of the steps in each embodiment. These steps or operations are only examples, and the embodiments of the present application can also perform other operations or variations of various operations. In addition, the various steps can be performed in different orders presented in the embodiments, and it is possible that not all operations in the embodiments of the present application are to be performed. Moreover, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0225] It should also be understood that in the present application, "when", "if" and "if" all mean that the device will take corresponding actions under certain objective circumstances, and do not limit the time, nor do they require that the device must have a judgment action when it is implemented, nor do they mean that there are other limitations.

[0226] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0227] It should be understood that in each embodiment of the present application, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0228] The above contents are optional embodiments of the technical solution of this application, and are not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made within the principles of this application shall be included in the protection scope of this application.

Claims

1. A method for reusing the memory of a neural network chip, characterized in that: include: Determine a target tensor, where the target tensor is a tensor that needs to be exchanged from the neural network chip memory to the non-neural network chip memory; Determine the life cycle of the target tensor according to a plurality of computing operations in which the target tensor participates, wherein the life cycle of the target tensor is greater than or equal to the sum of the durations of the plurality of computing operations, and the life cycle of the target tensor is less than the duration from the start time of the plurality of computing operations to the end time of the plurality of computing operations; The neural network chip memory is reused according to the life cycle of the target tensor.

2. The method according to claim 1, characterized in that The target tensor is a tensor that cannot be generated multiple times during the calculation process of the neural network.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Determine a plurality of tensors, each of which is a tensor that needs to be exchanged from the neural network chip memory to the non-neural network chip memory, and the plurality of tensors includes the target tensor; The reusing the neural network chip memory according to the life cycle of the target tensor includes: According to the life cycle of the plurality of tensors, the plurality of tensors are exchanged from the neural network chip memory to the non-neural network chip memory through a single exchange operation; and / or, According to the life cycles of the multiple tensors, the multiple tensors are exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

4. The method according to claim 3, characterized in that The reusing of the neural network chip memory according to the life cycle of the target tensor includes: exchanging the multiple tensors from the neural network chip memory to the non-neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors; the multiple tensors include a first tensor participating in a forward calculation operation of a first neural network layer; The determining of the plurality of tensors comprises: Determining the first tensor before a forward computation operation of the first neural network layer is completed; After the forward computation operation of the first neural network layer is completed and before the first tensor is exchanged, the method further includes: Perform a forward computation operation of a second neural network layer, where the second neural network layer is a neural network layer located after the first neural network layer.

5. The method according to claim 3 or 4, characterized in that: The reusing of the neural network chip memory according to the life cycle of the target tensor includes: exchanging the multiple tensors from the non-neural network chip memory to the neural network chip memory through a single exchange operation according to the life cycle of the multiple tensors; the multiple tensors include a first tensor participating in the reverse calculation operation of the first neural network layer; The determining of the plurality of tensors comprises: Determining the first tensor before a reverse calculation operation of a second neural network layer is completed, where the second neural network layer is a neural network layer located after the first neural network layer; The exchanging the plurality of tensors from the non-neural network chip memory to the neural network chip memory through a single exchange operation comprises: Before the reverse calculation operation of the second neural network layer is completed, the first tensor is exchanged from the non-neural network chip memory to the neural network chip memory through a single exchange operation.

6. The method according to any one of claims 3 to 5, characterized in that The non-neural network chip memory includes a first memory and a second memory, and the transmission rate of the first memory is higher than the transmission rate of the second memory; The exchanging the plurality of tensors from the neural network chip memory to the non-neural network chip memory comprises: exchanging tensors with shorter calculation intervals among the multiple tensors to the first memory; Exchange the tensor with the longer calculation interval among the multiple tensors to the second memory; The calculation interval is the interval time between the first time any tensor of the multiple tensors participates in the calculation operation and the last time any tensor of the multiple tensors participates in the calculation operation.

7. The method according to claim 6, characterized in that The tensor with a shorter calculation interval is the tensor corresponding to the neural network layer after the target neural network layer, and the tensor with a longer calculation interval is the tensor corresponding to the target neural network layer and the neural network layer before the target neural network layer.

8. The method according to claim 7, characterized in that The number of neural network layers after the target neural network layer is equal to the smaller one of the first value and the number of neural network layers, and the first value is the result of rounding down the ratio of the capacity of the first memory to the amount of data of the tensor to be exchanged corresponding to the single-layer neural network.

9. A device for reusing the memory of a neural network chip, characterized in that: include: Module for executing the method as claimed in any one of claims 1 to 8.

10. A device for reusing the memory of a neural network chip, characterized in that: include: A processor, configured to execute a computer program stored in the memory so that the apparatus performs the method according to any one of claims 1 to 8; A communication interface is coupled to the processor and is used to input or output information.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that When the computer program product is executed by a processor, the method according to any one of claims 1 to 8 is performed.

Images