Electronic device, method and medium for neural network model training processing

Abstract

The present application provides an electronic device, method and medium for processing neural network model training, wherein a central processing unit is configured to perform storage control on intermediate tensors in neural network model training: for a generated first intermediate tensor, determine whether the current storage usage rate of the memory of an artificial intelligence processing unit exceeds a first threshold, and if it is determined that it exceeds the first threshold, transfer the first intermediate tensor from the memory of the artificial intelligence processing unit to the memory of the central processing unit; and before using the first intermediate tensor for calculation in backward calculation, determine the storage location of the first intermediate tensor, and if it is determined that the storage location is the memory of the central processing unit, retrieve the first intermediate tensor from the memory of the central processing unit to the memory of the artificial intelligence processing unit for backward calculation. The present application can dynamically unload intermediate tensors to the memory of the central processing unit according to the storage occupancy of the artificial intelligence processing unit.

Description

Technical Field

[0001] Some embodiments of the present application specifically relate to an electronic device, method and medium for neural network model training processing. Background Art

[0002] The development of neural network models has enabled breakthrough progress in artificial intelligence in various application fields, but it has also brought challenges in hardware computing power, storage capacity, and energy consumption. Reasonable configuration solutions are needed to solve the problem of storage resource usage during the calculation and training of neural network models. Specifically, the training process of neural network models requires a large amount of video memory resources, which reduces the training speed, places higher requirements on hardware resources, and increases the difficulty of training network models. Summary of the invention

[0003] Embodiments of the present application provide an electronic device, method, and medium for processing neural network model training, which are used to reduce video memory occupancy during neural network model training.

[0004] According to one aspect of the present application, there is provided an electronic device for training and processing a neural network model, comprising: a central processing unit, comprising a first memory corresponding to the central processing unit; and an artificial intelligence processing unit, comprising a second memory corresponding to the artificial intelligence processing unit, wherein the central processing unit is configured to perform storage control on intermediate tensors in the neural network model training process, including: using the artificial intelligence processing unit to perform forward calculations on the neural network model; for a first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in the second memory; determining whether a current storage usage rate of the second memory exceeds a first threshold, and if it is determined that the first threshold is exceeded, transferring the first intermediate tensor to the first memory; using the artificial intelligence processing unit to perform backward calculations on the neural network model; and before using the first intermediate tensor for calculation in the backward calculation, determining the storage location of the first intermediate tensor, and if it is determined that the storage location is the second memory, directly using the first intermediate tensor by the artificial intelligence processing unit, and if it is determined that the storage location is the first memory, retrieving the first intermediate tensor from the first memory to the second memory for backward calculation.

[0005] According to some embodiments of the present application, the process of generating a first intermediate tensor in forward calculation corresponds to a first processing queue, the process of transferring the first intermediate tensor to a first memory corresponds to a second processing queue, the process of using the first intermediate tensor for calculation in backward calculation corresponds to a third processing queue, and the process of retrieving the first intermediate tensor to a second memory corresponds to a fourth processing queue, wherein the central processing unit performs storage control of the intermediate tensors in the neural network model training process further including: synchronously controlling the first processing queue and the second processing queue so that the second processing queue is executed after the first processing queue is executed; and synchronously controlling the third processing queue and the fourth processing queue so that the third processing queue is executed after the fourth processing queue is executed.

[0006] According to some embodiments of the present application, the central processing unit performs storage control on intermediate tensors in the neural network model training process and further includes: for the second intermediate tensor generated in the forward calculation, when determining to transfer the second intermediate tensor to the first memory, determining whether the current storage usage rate of the second memory exceeds the second threshold; and if it is determined that the second threshold is exceeded, synchronously controlling the fifth processing queue that generates other intermediate tensors in the forward calculation and the sixth processing queue that transfers the second intermediate tensor to the first memory, so as to suspend the fifth processing queue until the current storage usage rate does not exceed the second threshold, wherein the other intermediate tensors are one or more intermediate tensors different from the second intermediate tensor.

[0007] According to some embodiments of the present application, the central processing unit performs storage control on intermediate tensors in the neural network model training process, and further includes: in the process of using the third intermediate tensor for calculation in the backward calculation, when determining to retrieve the third intermediate tensor from the first memory to the second memory, determining whether the current storage usage rate of the second memory exceeds the third threshold value; and if it is determined that the third threshold value is exceeded, synchronously controlling the seventh processing queue that uses other intermediate tensors in the backward calculation and the eighth processing queue that retrieves the third intermediate tensor from the first memory to the second memory, so as to suspend the eighth processing queue until the current storage usage rate does not exceed the third threshold value, wherein the other intermediate tensors are one or more intermediate tensors different from the third intermediate tensor.

[0008] According to some embodiments of the present application, there are multiple artificial intelligence processing units, each artificial intelligence processing unit includes a second memory corresponding to it, and the central processing unit is configured to perform storage control of intermediate tensors for each artificial intelligence processing unit respectively.

[0009] According to some embodiments of the present application, the first intermediate tensor is an activation value tensor generated in the forward calculation.

[0010] According to some embodiments of the present application, there are multiple artificial intelligence processing units, each artificial intelligence processing unit includes a second memory corresponding to it, the first intermediate tensor is the activation value tensor generated in the forward calculation, and the central processing unit performs storage control on the intermediate tensors in the neural network model training process, and further includes: dividing the weight tensor calculated with the activation value tensor into N parts, and allocating them to N artificial intelligence processing units among the multiple artificial intelligence processing units, so that the calculation process of the activation value tensor and the weight tensor is executed in parallel by the N artificial intelligence processing units; when it is determined that the activation value tensor is transferred to the first memory, the activation value tensor is divided into N parts, and for the N artificial intelligence processing units, 1 / N parts of the activation value tensor are transferred to the first memory respectively; and before using the activation value tensor in the backward calculation, 1 / N parts of the activation value tensor are retrieved respectively, and the complete activation value tensor is obtained by collective communication operation.

[0011] According to some embodiments of the present application, the central processing unit implements storage control of intermediate tensors in the neural network model training process as a first instruction code implemented by a context manager, and the first instruction code is executed by the central processing unit, wherein the neural network model is implemented as a second instruction code decoupled from the first instruction code.

[0012] According to another aspect of the present application, a method for training and processing a neural network model is provided. The method is applicable to an electronic device, which includes a central processing unit and an artificial intelligence processing unit, the central processing unit includes a first memory corresponding to the central processing unit, and the artificial intelligence processing unit includes a second memory corresponding to the artificial intelligence processing unit. The method according to the present application includes: performing forward calculation on the neural network model; for the first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in the second memory; determining whether the current storage usage rate of the second memory exceeds the first threshold, and if it is determined that it exceeds the first threshold, transferring the first intermediate tensor to the first memory; performing backward calculation on the neural network model; and before using the first intermediate tensor for calculation in the backward calculation, determining the storage location of the first intermediate tensor, if it is determined that the storage location is the second memory, the artificial intelligence processing unit directly uses the first intermediate tensor, and if it is determined that the storage location is the first memory, retrieving the first intermediate tensor from the first memory to the second memory for backward calculation.

[0013] According to another aspect of the present application, a non-temporary computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the processor performs the steps of the method for neural network model training processing as described above.

[0014] By utilizing the electronic device, method, and medium for neural network model training processing provided in the embodiments of the present application, it is possible to dynamically unload intermediate tensors generated in the forward calculation of the neural network model to the system memory according to the memory usage rate of the artificial intelligence processing unit, so as to reduce the video memory occupancy generated during the neural network model training process, thereby enabling neural network model training to be implemented with limited artificial intelligence processing unit storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic block diagram of an electronic device according to an embodiment of the present application is shown;

[0017] Figure 2 A schematic system structure block diagram of an electronic device according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic flow chart of storage control performed by an electronic device according to an embodiment of the present application is shown;

[0019] Figure 4 A schematic flow chart of a method for training a neural network model according to an embodiment of the present application is shown;

[0020] Figure 5 A schematic block diagram of a computer-readable storage medium according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0022] In addition, as shown in the present application and the claims, unless the context clearly indicates an exception, the words "one", "a", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. The words "first", "second" and similar words used in the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, the words "include" or "comprise" and the like mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. In addition, it can be understood that the above-mentioned words such as "include" or "comprise" are not limited to being located inside it in a physical sense, but can also be expressed as a logical inclusion relationship. The words such as "connect" or "connected" and the like are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect, and can also represent control logic connection relationships.

[0023] Flowcharts are used in this application to illustrate the steps of the method according to the embodiments of the present application. It should be understood that the preceding or following steps are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes.

[0024] It is understood that the professional terms, nouns, etc. described herein have meanings known to those skilled in the art. To facilitate understanding of the present application, the terms that may be involved in at least some embodiments of the present application are explained as follows.

[0025] Electronic devices generally include a central processing unit (CPU), which is the core of device operation and control and the final execution unit for information processing and program running.

[0026] For the CPU, a multi-level storage structure is configured to provide storage support for the CPU to implement computing functions. First, the CPU can be configured with a multi-level cache architecture, such as a first-level cache, a second-level cache, and a third-level cache (L1 / L2 / L3), which are generally located inside the CPU chip. In addition, the CPU is also configured with system memory, or off-chip memory, such as dynamic random access memory (DRAM). Furthermore, the CPU can also be configured with external storage, such as a disk or a solid-state drive.

[0027] With the development of artificial intelligence and other fields, unprecedented challenges have been posed to the computing power of devices. Traditional CPU architectures, due to memory access bottlenecks and other reasons, are gradually unable to meet computing needs when processing complex deep learning models.

[0028] Graphics Processing Unit (GPU) has become the first choice for accelerating deep learning due to its highly parallel architecture and powerful floating-point computing capabilities. The parallel computing characteristics of GPU are highly consistent with the parallel computing nature of neural networks, enabling it to show excellent performance when processing large-scale matrix operations and neural network models. With the continuous development of deep learning algorithms, GPU has been widely used in image recognition, natural language processing, computer vision and other fields, providing strong computing power support for the development of artificial intelligence. As one of the key hardware components of a computer system, GPU can be embedded and integrated into a chipset (SoC) or connected to the motherboard as an independent independent graphics unit.

[0029] The GPU storage system consists of multiple levels, from video memory to faster cache memory (generally located inside the GPU chip), such as L1 / L2 cache. These hierarchical structures provide different access speeds and capacities to meet various computing and graphics processing needs.

[0030] As an example, the video memory of a GPU may be implemented as Graphics Double Data Rate Synchronous Dynamic Random Access Memory (GDDRSDRAM), which is a storage medium for graphics cards.

[0031] As another example, the GPU's video memory can be implemented as High Bandwidth Memory (HBM), which is another commonly used video memory medium. The storage medium is stacked in a three-dimensional design space. This three-dimensional structure is closer to the chip unit and occupies a relatively small space. This design has improved the bandwidth and storage capacity of HBM, mainly because as the GPU becomes more powerful, it is necessary to access data from the memory faster and shorten the application processing time.

[0032] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] In artificial intelligence applications, a variety of neural network models have been developed for different application fields. Generally, as application processing becomes more and more complex, the model parameters of the neural network model are also increasing. Generally, neural network models with parameters reaching hundreds of millions or even billions or hundreds of billions can be called large models, such as the GPT series models (Generative Pre-Trained Transformer, GPT) and LLaMA (Large Language Model MetaAI) models in the field of natural language processing. Large models can better understand natural language, perform smooth text generation, understanding and multimodal task processing, and provide users with a more natural and efficient human-computer interaction experience.

[0034] In deep learning, training neural networks is a crucial process, which optimizes the parameters of the neural network model through repeated iterations so that it can better adapt to tasks and data. A single iteration in the neural network model training process can include four stages: forward calculation, loss calculation, backward calculation, and parameter update.

[0035] In the forward calculation, the input data can be fed into the neural network model, and the neurons in each layer will be calculated according to their input and weights, and the output will be obtained through the activation function. This process starts from the input layer and goes backward layer by layer until the output layer obtains the final prediction result. In addition, various intermediate results will be generated during the forward calculation, such as activation values, gradient parameters, optimizer status, etc. During the training process, some activation values ​​need to be saved for backward calculation.

[0036] In the backward calculation, at this stage, for example, the parameters of the neural network model can be updated by the gradient descent method so that the loss value obtained based on the loss function is minimized, thereby achieving the training purpose of the neural network model.

[0037] Tensor is a multidimensional array data structure. In the field of deep learning, the dimensions of input data, intermediate data processed during deep learning, and output data are flexible and not exact, so a flexible data form is needed to describe various types of data, which gives rise to the concept of tensor. In the field of deep learning, all data to be calculated are stored and exist in the form of tensors. If the data is not in the form of tensors, it needs to be converted into the data structure of tensors first. For example, the above activation value can be described as an activation value tensor for various calculation processes.

[0038] Tensor Parallelism (TP) means splitting a large tensor (such as a weight tensor) in an operation into several smaller tensors and distributing these split tensors to different devices (for example, multiple GPUs) for parallel computing.

[0039] In Python programming, a context manager is a special object that defines the actions that should be performed when entering and exiting a code block. Context managers are usually used with the with statement for resource management, such as opening and closing files, acquiring and releasing locks, etc. The context manager is a powerful management mechanism for managing the acquisition and release of resources. It provides a concise and safe way to handle the opening, closing, and exception handling of resources, making the code more readable and maintainable, while enhancing the robustness of the program.

[0040] The development of neural network models has enabled breakthrough progress in artificial intelligence in various application fields, but it has also brought challenges in hardware computing power, storage resources, and energy consumption. Reasonable configuration solutions are needed to solve the problem of storage resource usage during the calculation and training of neural network models. Specifically, the training process of neural network models requires a large amount of video memory resources, which reduces the training speed, places high demands on hardware resources, and increases the difficulty of training network models. Especially for the above-mentioned large models, the demand for video memory usage is huge. For example, the main data that occupies video memory during large model training includes model parameters, input data, intermediate calculation results, activation values, optimizer status, gradient parameters, etc.

[0041] The related art involves recalculation technology, which is used to control the occupation of GPU video memory by activation value tensors. Specifically, it selectively stores only a part of activation values ​​in GPU video memory during forward calculation as activation value checkpoints. For activation values ​​that are not stored, another forward calculation is required during backward calculation to obtain the above-mentioned unstored activation values. However, since the recalculation introduces an additional recalculation process, it will inevitably lead to a significant decline in training performance.

[0042] In response to the above technical problems, the present application proposes that the CPU in the device uniformly manage the intermediate tensors generated during the training process of the neural network model. When the GPU video memory occupancy is too high, some of the intermediate tensors are unloaded to the system memory, thereby solving the video memory occupancy problem during the neural network training process. This is especially beneficial for large model training, thereby enabling neural network model training to be implemented under limited GPU video memory hardware conditions, while also ensuring that the training performance is not affected.

[0043] Next, the implementation scheme of the electronic device for neural network model training processing according to the embodiment of the present application will be described in detail with reference to the accompanying drawings.

[0044] Figure 1 A schematic flow chart of an electronic device for processing a neural network model training according to an embodiment of the present application is shown. Figure 1 As shown, the electronic device 100 according to the embodiment of the present application may be a device including both a central processing unit (CPU) 110 and an artificial intelligence processing unit (for example, the above-mentioned GPU) 120, which may also be referred to as a computing device. Specifically, the central processing unit may include a corresponding first memory 111, and the artificial intelligence processing unit may include a corresponding second memory 121. As an example, the first memory 111 may refer to the above-mentioned system memory, or may be referred to as off-chip memory, memory, etc., for example, implemented as DRAM, which is not limited here. As an example, the second memory 121 may refer to the above-mentioned video memory, for example, implemented in the form of GDDR SDRAM or HBM, etc., which is not limited here. It can be understood that the number of artificial intelligence processing units in the embodiment of the present application may be one or more, which will be described in the following embodiments.

[0045] As some implementations, the artificial intelligence processing unit in this article can be any one of a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), an NPU (Neural Network Processing Unit), a DPU (Deep Learning Processing Unit), an APU (Accelerated Processing Unit), and a GPGPU (General-Purpose computing on Graphics Processing Unit), without limitation. In some specific descriptions below, the artificial intelligence processing unit is described as a GPU as a specific example, and it is understood that the solution of the present application is not limited to this.

[0046] The neural network model mentioned in this article can be, for example, a natural language processing neural network, or other neural network models for natural language processing, for example, it can be a Transformer network model, a ChatGenerative Pre-trained Transformer (Ghat-GPT) network model, etc. As another example, the neural network model can be the above-mentioned LLaMA model. It is understandable that the equipment and methods provided by this application can be applicable to the training process of various existing neural network models or new models that may appear in future developments, and are not limited here.

[0047] According to an embodiment of the present application, the central processing unit CPU can be configured to perform storage control on the intermediate tensors generated by the GPU during the neural network model training process. The storage control can be understood as the CPU in the device uniformly managing the intermediate tensors generated during the neural network model training process. When the GPU video memory is insufficient, some of the intermediate tensors are unloaded to the system memory, thereby solving the video memory occupation problem during the neural network training process.

[0048] According to an embodiment of the present application, the above-mentioned storage control includes: using an artificial intelligence processing unit to perform forward calculations on a neural network model; for a first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in a second memory; determining whether a current storage usage rate of the second memory exceeds a first threshold, and if it is determined that the first threshold is exceeded, transferring the first intermediate tensor to the first memory; using an artificial intelligence processing unit to perform backward calculations on the neural network model; and before using the first intermediate tensor for calculation in the backward calculation, determining the storage location of the first intermediate tensor, and if it is determined that the storage location is the second memory, directly using the first intermediate tensor by the artificial intelligence processing unit, and if it is determined that the storage location is the first memory, retrieving the first intermediate tensor from the first memory to the second memory for backward calculation.

[0049] In the process of training a neural network model using an electronic device according to an embodiment of the present application, an artificial intelligence processing unit (such as a GPU) first performs forward calculations based on the parameters of the neural network model, wherein the samples used for training can be input into the neural network model, and the model structure layer by layer is calculated according to its input and weight, and the training output result is obtained. Some intermediate results will be generated in the forward calculation, such as the above-mentioned activation value tensor, which will be used in the subsequent backward calculation. Therefore, these intermediate results need to be saved in the video memory for subsequent use, that is, for the first intermediate tensor generated in the forward calculation, the first intermediate tensor is stored in the second memory. It can be understood that the first intermediate variable in this article can be any intermediate result generated in the forward calculation and needs to be saved to the video memory, without limitation. According to the rules for the artificial intelligence processing unit to process the neural network model, these intermediate results need to be saved in the video memory together with other parameters of the neural network model.

[0050] As an example, the parameters of the neural network model training process are stored in the video memory according to the model's hierarchy. For a neural network model, it is generally composed of a multi-level network structure. Especially for a large model, the number of network levels can reach dozens or even hundreds. The various parameters and intermediate results that need to be saved to the video memory are stored according to the hierarchy.

[0051] After storing the first intermediate tensor in the video memory, the CPU can then judge the occupancy rate of the video memory of the artificial intelligence processing unit. If the video memory occupancy is too high, affecting the operating efficiency of the artificial intelligence processing unit, the first intermediate tensor can be transferred to the system memory to reduce the occupancy of the video memory. According to an embodiment of the present application, it can be determined whether the current storage usage rate of the video memory exceeds the first threshold. If it is determined that it exceeds the first threshold, the first intermediate tensor is transferred to the memory. As an example, the first threshold can be set to 0.7, that is, when it is determined that the video memory occupancy rate exceeds 70%, the intermediate tensor is transferred to the memory. It can be understood that the transfer here refers to transferring the tensor from the video memory of the artificial intelligence processing unit to the memory via a bus, that is, the data of the tensor is no longer stored in the video memory.

[0052] As an implementation, the frequency of determining the video memory occupancy by the CPU may be performed tensor by tensor. For example, after each intermediate tensor is stored in the video memory, the video memory occupancy rate is judged once, and whether it needs to be transferred to the internal memory is determined based on the judgment result. As another implementation, the frequency of determining the video memory occupancy by the CPU may be performed according to a predetermined number of tensors. For example, after S intermediate tensors are stored in the video memory, the video memory occupancy rate is judged once, and whether one or more of the S intermediate tensors are transferred to the internal memory is determined based on the judgment result, where S may be an integer greater than or equal to 2. As other implementations, the above judgment may also be performed at predetermined time intervals, which are not limited here.

[0053] It is understandable that after the forward calculation is completed, part of the intermediate results generated in the forward calculation are stored in the video memory, and another part has been unloaded from the video memory to the internal memory. Therefore, according to the storage control scheme provided in the embodiment of the present application, the CPU in the device can uniformly control the storage of the intermediate vectors generated in the GPU calculation process, dynamically adjust the storage capacity of the GPU video memory, and when the GPU video memory is overloaded, timely unload part of the intermediate tensors to the system memory, thereby avoiding excessive video memory occupation, thereby ensuring the computing power of the neural network calculation process and reducing the hardware requirements for the video memory.

[0054] Next, in the backward calculation, due to the above-mentioned transfer step, before using the intermediate variable for calculation, it is necessary to determine the storage location of the intermediate variable. If the storage location is the video memory, it can be directly read and used by the GPU. If the storage location is the memory, the intermediate tensor needs to be retrieved from the memory to the video memory first, and then the GPU reads the intermediate tensor from the video memory for backward calculation.

[0055] As an implementation method, when the CPU performs the above-mentioned storage control, each intermediate tensor generated can be numbered, and the intermediate variable can be uniquely identified according to the number. In addition, during the storage control process, the storage location of the corresponding intermediate variable can also be recorded according to the number. For example, the intermediate variable that has been transferred to the memory is identified, and the identification can indicate whether the storage location of the intermediate variable is in the video memory or the memory. In addition, as an example, the intermediate tensor generated in the forward calculation itself includes both specific data and some parameter information, such as the size of the tensor, the corresponding GPU information, the neural network level corresponding to the tensor, etc. These parameter information are also managed uniformly through the storage control of the CPU.

[0056] According to some embodiments of the present application, the first intermediate tensor mentioned above can be an activation value tensor generated in the forward calculation, that is, using the electronic device provided according to the implementation of the present application, the CPU implements the unified management of the activation value tensor during the training of the neural network model, so as to avoid the problem of excessive occupation of the video memory of the artificial intelligence processing unit by the activation value tensor, which affects the computing efficiency. In the above manner, the stored activation value tensor is decoupled from other parameters stored in the neural network model according to the network model level, and is identified and uniformly managed by the CPU, for example, through the above-mentioned numbering. Further, according to the storage control of the activation value tensor in the embodiment of the present application, more flexible dynamic adjustment can be achieved, and the adjustment granularity of the activation value tensor is finer. For example, it is possible to determine whether the video memory occupancy exceeds the threshold value for each tensor. If it exceeds, the activation value tensor can be unloaded to the memory only, thereby achieving flexible and fine-grained storage control.

[0057] It can be understood that, in some other embodiments of the present application, the above-mentioned first intermediate tensor can also be a tensor other than the activation value tensor generated in the forward calculation, which is not limited here, and the implementation process is similar.

[0058] According to some embodiments of the present application, the number of artificial intelligence processing units in the electronic device is multiple, and the multiple in this article can be understood as two or more. Specifically, each artificial intelligence processing unit includes a second memory corresponding to it, and the central processing unit is configured to perform storage control of intermediate tensors for each artificial intelligence processing unit separately. In actual application, the device is generally configured with multiple GPUs for parallel computing. For example, in the training process of a neural network model, multiple GPUs perform parallel processing. In the CPU, each GPU corresponds to a separate process, so that the CPU can separately and independently perform the above-mentioned storage control process of each GPU video memory in the separate process of each GPU.

[0059] Figure 2A schematic system structure block diagram of an electronic device according to an embodiment of the present application is shown, including a case where the electronic device includes multiple artificial intelligence processing units. Specifically, Figure 2 In this paper, the artificial intelligence processing unit is taken as a GPU as an example. Figure 2 As shown, the electronic device 100 may include a CPU 110, which may include a CPU core and a multi-level cache architecture for the CPU, such as L1 / L2 / L3 cache, and may also be configured with a system memory 111_1. In addition, the electronic device may also be configured with multiple GPUs. Figure 2 Only two GPUs are schematically shown, including GPU 1 numbered 120_1 and GPU 2 numbered 120_2. The internal configuration of each GPU can be the same. Taking GPU 1 as an example, it is configured with a GPU core, a cache architecture for the GPU, and a GPU memory 121_1. As shown above, in the CPU, each GPU corresponds to a separate process, so that the CPU can separately and independently perform the above storage control process of each GPU memory in the separate process of each GPU.

[0060] According to some embodiments of the present application, the process of generating the first intermediate tensor in the forward calculation corresponds to the first processing queue, the process of transferring the first intermediate tensor to the first memory corresponds to the second processing queue, the process of using the first intermediate tensor for calculation in the backward calculation corresponds to the third processing queue, and the process of retrieving the first intermediate tensor to the second memory corresponds to the fourth processing queue. According to an embodiment of the present application, the process of forward calculation and intermediate tensor transfer is performed asynchronously, that is, belongs to different processing queues. Similarly, the backward calculation and the process of retrieving the intermediate tensor from the memory are also performed asynchronously, which separates the storage control process from the GPU calculation process and performs independently without interfering with each other. Among them, the concept of the above queue can represent a stream or OP in the GPU field, for example.

[0061] According to some embodiments of the present application, the storage control of the intermediate tensors in the neural network model training process by the central processing unit also includes: synchronous control of the first processing queue and the second processing queue, so that the second processing queue is executed after the first processing queue is executed; and synchronous control of the third processing queue and the fourth processing queue, so that the third processing queue is executed after the fourth processing queue is executed. As described above, in the embodiments of the present application, the forward calculation and the process of transferring the intermediate tensors are performed asynchronously, that is, they belong to different processing queues. Similarly, the backward calculation and the process of retrieving the intermediate tensors from the memory are also performed asynchronously and belong to different processing queues. On the basis of performing the above operations asynchronously, the present application proposes to synchronously control the associated processing queues to ensure that the associated processing queues are executed in a logical order.

[0062] For example, the above-mentioned synchronous control includes executing the transfer processing task after the processing task of generating the intermediate tensor in the forward calculation is completed, so as to avoid data errors caused by transferring the intermediate dynamic results to the memory. Similarly, the above-mentioned synchronous control also includes the need to ensure that the corresponding calculation task is performed after the intermediate tensor retrieval task is completed during the backward calculation. Therefore, the data stability and accuracy of the transfer and retrieval steps of the intermediate tensor can be guaranteed through the above synchronous control, such as avoiding the execution of the transfer processing task before the execution of the processing task of generating the intermediate tensor is completed, resulting in the transfer data being an erroneous intermediate result. For another example, avoiding the execution of the corresponding calculation task before the intermediate tensor retrieval task is completed, resulting in the corresponding calculation task producing an erroneous calculation result due to lack of data.

[0063] According to some embodiments of the present application, the central processing unit performs storage control on intermediate tensors in the neural network model training process and further includes: for the second intermediate tensor generated in the forward calculation, when determining to transfer the second intermediate tensor to the first memory, determining whether the current storage usage rate of the second memory exceeds the second threshold; and if it is determined that the second threshold is exceeded, synchronously controlling the fifth processing queue that generates other intermediate tensors in the forward calculation and the sixth processing queue that transfers the second intermediate tensor to the first memory, so as to suspend the fifth processing queue until the current storage usage rate does not exceed the second threshold, wherein the other intermediate tensors are one or more intermediate tensors different from the second intermediate tensor.

[0064] In actual application, there is a difference in the implementation speed of the computing task and the processing task of transferring the intermediate tensors. In the forward calculation, for example, some processing tasks are executed faster due to the small amount of calculation, which is faster than the speed of the above-mentioned video memory unloading. In this process, more intermediate tensors that need to be unloaded will be generated, which makes it impossible to execute synchronously with the video memory unloading. For example, 10 intermediate tensors may be generated in the computing task, but the unloading task only executes the unloading of 2 intermediate tensors due to its relatively slow speed. In this case, the CPU's storage control can judge the current video memory occupancy rate again. If it exceeds the second threshold, the fifth processing queue that generates other intermediate tensors in the forward calculation can be controlled to suspend until the current storage usage rate does not exceed the second threshold, thereby avoiding the unloading blocking problem caused by the above-mentioned inconsistent execution speed. As an example, the above-mentioned second threshold can be set to be equal to the above-mentioned first threshold, or it can also be set to other values, such as greater than the above-mentioned first threshold.

[0065] According to some embodiments of the present application, the central processing unit performs storage control on intermediate tensors in the neural network model training process, and further includes: in the process of using the third intermediate tensor for calculation in the backward calculation, when determining to retrieve the third intermediate tensor from the first memory to the second memory, determining whether the current storage usage rate of the second memory exceeds the third threshold value; and if it is determined that the third threshold value is exceeded, synchronously controlling the seventh processing queue that uses other intermediate tensors in the backward calculation and the eighth processing queue that retrieves the third intermediate tensor from the first memory to the second memory, so as to suspend the eighth processing queue until the current storage usage rate does not exceed the third threshold value, wherein the other intermediate tensors are one or more intermediate tensors different from the third intermediate tensor.

[0066] Similar to the asynchrony that may occur in the forward calculation due to the mismatch between the speed of the computing task and the unloading task, in the backward calculation, there may be a slow execution speed of the processing of the computing task, while the speed of retrieving activation value tensors such as activation value tensors is relatively fast. In this case, it is possible that 10 activation value tensors have been retrieved to the video memory in the retrieval task, while the actual computing task only uses 2 retrieved activation value tensors. At this time, if synchronization control is not performed, the video memory may be occupied too high. In this case, the storage control of the CPU can suspend the retrieval of the third intermediate tensor from the first memory to the eighth processing queue of the second memory until the current storage usage rate does not exceed the third threshold, that is, suspend the retrieval task of some intermediate tensors to wait for the execution of the computing task. As an example, the above-mentioned third threshold can be set to be equal to the above-mentioned second threshold or the first threshold, or it can also be set to other values, which are not limited here. According to an embodiment of the present application, the specific values ​​of the above-mentioned first threshold, second threshold and third threshold can be dynamically set according to actual application scenarios and requirements.

[0067] According to some embodiments of the present application, there are multiple artificial intelligence processing units, each artificial intelligence processing unit includes a second memory corresponding thereto, and the first intermediate tensor is an activation value tensor generated in the forward calculation. Specifically, the central processing unit performs storage control on the intermediate tensors in the training process of the neural network model, and further includes: dividing the weight tensor calculated with the activation value tensor into N parts, and allocating them to N artificial intelligence processing units among the multiple artificial intelligence processing units, so that the calculation process of the activation value tensor and the weight tensor is performed in parallel by the N artificial intelligence processing units; when it is determined that the activation value tensor is transferred to the first memory, the activation value tensor is divided into N parts, and for the N artificial intelligence processing units, 1 / N parts of the activation value tensor are transferred to the first memory respectively; and before using the activation value tensor in the backward calculation, 1 / N parts of the activation value tensor are retrieved respectively, and the complete activation value tensor is obtained by collective communication operation.

[0068] As an example, for example, in the process of training a large model, the tensor parallelism (TP) technology may be applied, wherein multiple GPUs in the device can be divided into multiple TP groups, each TP group corresponds to multiple GPUs, and as an example, a TP group is composed of 4 GPUs, and the GPUs in the TP group can perform parallel processing for tensors with large data volumes. As an example, assuming that the computing task is tensor A multiplied by tensor B, where tensor A can be an activation value tensor and tensor B can be a weight tensor, in this case, tensor B can be split, for example, into 4 smaller tensors, so that the 4 GPUs in the TP group can perform 1 / 4 of the computing task of the overall computing task of tensor A multiplied by tensor B respectively, and the tensor computing task with a large computing amount can be realized through parallel computing within the TP group. For the 4 GPUs in the TP group, which include the complete tensor A and the 1 / 4 tensor B allocated to it for processing, after the GPUs complete their respective computing tasks, the computing results of each GPU are spliced ​​into the final computing result, that is, the computing result of tensor A multiplied by tensor B.

[0069] In view of the above situation, when implementing the storage control according to the embodiment of the present application, the storage control of each GPU is performed separately. In addition, if it is determined that the above-mentioned A tensor needs to be unloaded, considering that the A tensors stored on the 4 GPUs in the TP group are actually exactly the same, when performing the unloading process of the activation value tensor, the A tensor can also be similarly divided, that is, it is also divided into 4 parts according to the number of GPUs in the TP group, so that only one part can be unloaded for each GPU. Correspondingly, in the process of backward calculation, since the calculation of each GPU requires the use of a complete A tensor, an additional operation is required at this time, that is, the complete activation value tensor is obtained by using the collective communication operation (allgather). In other words, after each GPU in the TP group retrieves this part of the A tensor on other GPUs, it uses allgather to piece together a complete A tensor, and then performs the corresponding backward calculation process.

[0070] According to some embodiments of the present application, the storage control of the intermediate tensors in the training process of the neural network model by the central processing unit is implemented as a first instruction code implemented by a context manager, and the first instruction code is executed by the central processing unit, wherein the neural network model is implemented as a second instruction code decoupled from the first instruction code. For example, the storage control function provided according to the embodiment of the present application is implemented by the Context Manager in Python programming. As an example, it is implemented as a separate storage control code. It can be understood that the storage control code is independent of the code that implements the neural network model, that is, it is decoupled from the model code, which enables the storage control provided according to the embodiment of the present application to be quickly adapted to various neural network models. On the basis of the original neural network model code, an additional storage control code is added to implement the storage control function of the present application. Even if the model parameters change, it is not affected by it, thereby providing a fast and stable video memory occupancy control function.

[0071] As a specific implementation example, Figure 3 FIG. 1 shows a schematic flow chart of storage control performed by an electronic device according to an embodiment of the present application. Figure 3 To describe the implementation process of storage control provided according to the embodiment of the present application. Figure 3 In the description, the control logic for storage control of the activation value tensor generated by the GPU executed by the CPU is represented as an activation value manager. As an example, the activation value manager can be implemented as the above-mentioned storage control code (PythonContext Manager) executed by the CPU, which is used to manage and control the storage of the activation value tensor generated during the training process of the neural network model, that is, whether it is stored in the video memory or the main memory, so as to dynamically adjust the GPU video memory occupancy problem.

[0072] like Figure 3 As shown, first, in the forward calculation process, in step S1, the generated activation value tensor will be stored in the GPU memory via the activation value manager. In this process, the manager can, for example, manage the parameter information of the activation value tensor and number it for unified management, such as recording its storage location, tensor size, corresponding GPU, etc. Then, in the backward calculation process, in step S2, the activation value tensor is obtained via the activation value manager for backward calculation.

[0073] Specifically, in step S3, for the activation value tensor stored in the video memory in the forward calculation, the activation value manager can also determine the current storage usage of the video memory. For example, if it exceeds 70%, the activation value tensor is controlled to be unloaded from the GPU video memory to the system memory to reduce the occupation of the video memory and ensure the computational efficiency of the GPU computing task. In step S4, for the activation value tensor that has been unloaded to the system memory, before the computing task of the backward calculation uses the activation value tensor, the activation value manager can control the activation value tensor to be retrieved from the system memory to the GPU video memory for backward calculation.

[0074] In addition, if Figure 3 As shown, in the process of unloading and retrieving the activation value tensor, you can also refer to the method described above to perform synchronous control and tensor splitting to ensure the accuracy and flexibility of the unloading task. The specific execution process of synchronous control and tensor splitting can refer to the above description and will not be repeated here.

[0075] An electronic device that utilizes the neural network model training processing provided by the embodiments of the present application can dynamically unload the intermediate tensors generated in the forward calculation of the neural network model to the system memory according to the memory usage of the artificial intelligence processing unit (e.g., GPU), so as to reduce the video memory occupancy generated during the neural network model training process, thereby enabling the neural network model training to be implemented with limited artificial intelligence processing unit storage resources.

[0076] According to another aspect of the present application, a method for training a neural network model is provided. The method is applicable to an electronic device, the electronic device comprising a central processing unit and an artificial intelligence processing unit, the central processing unit comprising a first memory corresponding to the central processing unit, and the artificial intelligence processing unit comprising a second memory corresponding to the artificial intelligence processing unit.

[0077] Figure 4 FIG. 1 is a schematic flow chart of a method for training a neural network model according to an embodiment of the present application. Figure 4 As shown, the method for training the neural network model according to the present application includes the following steps:

[0078] Step S401: performing forward calculation on the neural network model;

[0079] Step S402: for a first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in a second memory;

[0080] Step S403: determining whether the current storage usage rate of the second memory exceeds a first threshold, and if it is determined that the current storage usage rate exceeds the first threshold, transferring the first intermediate tensor to the first memory;

[0081] Step S404: performing backward calculation on the neural network model; and

[0082] Step S405: Before using the first intermediate tensor for calculation in the backward calculation, determine the storage location of the first intermediate tensor. When the storage location is determined to be the second memory, the first intermediate tensor is directly used by the artificial intelligence processing unit. When the storage location is determined to be the first memory, the first intermediate tensor is retrieved from the first memory to the second memory for backward calculation.

[0083] Using the method according to the embodiment of the present application, in the process of training the neural network model, the GPU first performs forward calculation based on the neural network model parameters, wherein the sample for training can be input into the neural network model, and the model structure layer by layer is calculated according to its input and weight, and the training output result is obtained. Some intermediate results will be generated in the forward calculation, such as the above-mentioned activation value tensor, which will be used in the subsequent backward calculation. These intermediate results need to be saved in the video memory for subsequent use, that is, for the first intermediate tensor generated in the forward calculation, the first intermediate tensor is stored in the second memory. It can be understood that the first intermediate variable in this article can be any intermediate result generated in the forward calculation and needs to be saved to the video memory, which is not limited here. After the first intermediate tensor is stored in the video memory, the CPU can then judge the occupancy rate of the GPU video memory. If the video memory occupancy is too high, affecting the GPU operation efficiency, the first intermediate tensor can be transferred to the system memory to reduce the occupancy of the video memory. As an example, the first threshold can be set to 0.7, that is, when it is determined that the video memory occupancy rate exceeds 70%, the intermediate tensor is transferred to the memory. It can be understood that the transfer here refers to transferring the tensor from the GPU video memory to the memory via a bus, that is, the data of the tensor is no longer stored in the video memory.

[0084] Next, in the backward calculation, due to the above-mentioned transfer step, before using the intermediate variable for calculation, it is necessary to determine the storage location of the intermediate variable. If the storage location is the video memory, it can be directly read and used by the GPU. If the storage location is the memory, the intermediate tensor needs to be retrieved from the memory to the video memory first, and then the GPU reads the intermediate tensor from the video memory for backward calculation.

[0085] According to some embodiments of the present application, the process of generating the first intermediate tensor in the forward calculation corresponds to the first processing queue, the process of transferring the first intermediate tensor to the first memory corresponds to the second processing queue, the process of using the first intermediate tensor for calculation in the backward calculation corresponds to the third processing queue, and the process of retrieving the first intermediate tensor to the second memory corresponds to the fourth processing queue. The method according to the embodiment of the present application also includes: synchronously controlling the first processing queue and the second processing queue so that the second processing queue is executed after the first processing queue is executed; and synchronously controlling the third processing queue and the fourth processing queue so that the third processing queue is executed after the fourth processing queue is executed.

[0086] The above-mentioned synchronous control can ensure the data stability and accuracy of the intermediate tensor transfer and retrieval steps. In the forward calculation, the transfer processing task is executed after the processing task of generating the intermediate tensor is completed, so as to avoid data errors caused by transferring the intermediate dynamic results to the memory. Similarly, in the backward calculation, it is necessary to ensure that the corresponding calculation task is performed after the intermediate tensor retrieval task is completed.

[0087] According to some embodiments of the present application, the method for neural network model training processing also includes: for the second intermediate tensor generated in the forward calculation, when determining to transfer the second intermediate tensor to the first memory, determining whether the current storage usage rate of the second memory exceeds a second threshold; and if it is determined that it exceeds the second threshold, synchronously controlling the fifth processing queue that generates other intermediate tensors in the forward calculation and the sixth processing queue that transfers the second intermediate tensor to the first memory, so as to suspend the fifth processing queue until the current storage usage rate does not exceed the second threshold, wherein the other intermediate tensors are one or more intermediate tensors different from the second intermediate tensor.

[0088] The above-mentioned synchronous control process can dynamically adjust the processing tasks according to the situation that the execution speed of the forward computing task and the tensor unloading task speed do not match each other in the actual application process. When it is determined that the video memory usage exceeds the second threshold, the processing queue for generating other intermediate tensors in the forward calculation can be controlled to be suspended until the current storage usage rate does not exceed the second threshold, thereby avoiding the unloading blocking problem caused by the above-mentioned inconsistent execution speed.

[0089] According to some embodiments of the present application, the method for neural network model training processing also includes: in the process of using the third intermediate tensor for calculation in the backward calculation, when determining to retrieve the third intermediate tensor from the first memory to the second memory, determining whether the current storage usage rate of the second memory exceeds a third threshold; and if it is determined that it exceeds the third threshold, synchronously controlling the seventh processing queue that uses other intermediate tensors in the backward calculation and the eighth processing queue that retrieves the third intermediate tensor from the first memory to the second memory, so as to suspend the eighth processing queue until the current storage usage rate does not exceed the third threshold, wherein the other intermediate tensors are one or more intermediate tensors different from the third intermediate tensor.

[0090] The above-mentioned synchronous control process can dynamically adjust the processing tasks according to the situation that the execution speed of the backward computing task and the tensor retrieval task speed do not match each other in the actual application process. When it is determined that the video memory usage exceeds the third threshold, the processing queue for retrieving intermediate tensors from the internal memory to the video memory can be controlled to be suspended until the current storage usage rate does not exceed the third threshold.

[0091] According to some embodiments of the present application, there are multiple artificial intelligence processing units, each artificial intelligence processing unit includes a second memory corresponding thereto, and storage control of intermediate tensors is performed for each artificial intelligence processing unit. In actual application, a device is generally configured with multiple GPUs for parallel computing. For example, in the training process of a neural network model, multiple GPUs perform parallel processing. In the CPU, each GPU corresponds to a separate process, so that the CPU can separately and independently perform the above-mentioned storage control process of each GPU video memory in the separate process of each GPU.

[0092] According to some embodiments of the present application, the first intermediate tensor is an activation value tensor generated in the forward calculation. That is, by using the method provided according to the implementation of the present application, the CPU implements unified management of activation value tensors during the training of the neural network model, thereby avoiding the problem of excessive occupation of GPU video memory by activation value tensors, which affects the computing efficiency.

[0093] According to some embodiments of the present application, there are multiple artificial intelligence processing units, each artificial intelligence processing unit includes a second memory corresponding to it, the first intermediate tensor is an activation value tensor generated in the forward calculation, and the method for neural network model training processing also includes: dividing the weight tensor calculated with the activation value tensor into N parts, and allocating them to N artificial intelligence processing units among the multiple artificial intelligence processing units, so that the calculation process of the activation value tensor and the weight tensor is executed in parallel by the N artificial intelligence processing units; when it is determined to transfer the activation value tensor to the first memory, the activation value tensor is divided into N parts, and for the N artificial intelligence processing units, 1 / N parts of the activation value tensor are transferred to the first memory respectively; and before using the activation value tensor in the backward calculation, 1 / N parts of the activation value tensor are retrieved respectively, and the complete activation value tensor is obtained by collective communication operation.

[0094] According to some embodiments of the present application, the method for training a neural network model is implemented as a first instruction code implemented by a context manager, the first instruction code is executed by a central processing unit, wherein the neural network model is implemented as a second instruction code decoupled from the first instruction code. For example, the storage control function provided according to the embodiment of the present application is implemented by a Context Manager in Python programming, and as an example, is implemented as a separate storage control code.

[0095] The steps of the method for training a neural network model provided in some embodiments of the present application can refer to the execution process of the electronic device for training a neural network model provided in the present application described above in combination with the accompanying drawings, and will not be described again here. Using the method provided in some embodiments of the present application, similar technical effects can be achieved.

[0096] According to yet another aspect of the present application, a non-transitory computer-readable storage medium is also provided. Figure 5 Schematic diagram of a computer-readable storage medium according to some embodiments of the present application is shown. Figure 5As shown, computer-readable instructions 501 are stored on the computer-readable storage medium 500. When the computer-readable instructions 501 are executed by a processor (e.g., a CPU in a device), the method of neural network model training processing described with reference to the above figures can be executed. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. For example, the computer-readable storage medium 500 may be connected to a computing device such as a computer, and then, when the computing device runs the computer-readable instructions 501 stored on the computer-readable storage medium 500, the method of neural network model training processing provided according to the embodiment of the present application as described above may be performed.

[0097] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of a software functional module. The present application is not limited to any particular form of combination of hardware and software.

[0098] Some embodiments of the present application provide an electronic device, method and medium for neural network model training processing, which are used to reduce the video memory occupancy of an artificial intelligence processing unit during the neural network model training process.

[0099] By utilizing the electronic device, method, and medium for neural network model training processing provided in the embodiments of the present application, it is possible to dynamically unload the intermediate tensors generated in the forward calculation of the neural network model to the system memory according to the memory usage rate of the artificial intelligence processing unit, so as to reduce the video memory occupancy generated during the neural network model training process, thereby enabling neural network model training to be implemented with limited artificial intelligence processing unit storage resources.

[0100] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0101] In addition, although the present application makes various references to certain units in computing devices and computing apparatuses according to embodiments of the present application, any number of different units may be used and run on the client and / or server. The units are merely illustrative, and different aspects of the parallel computing method, processor, electronic device, and computing device may use different units.

[0102] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0103] The above is an explanation of the present application and should not be considered as a limitation thereof. Although several exemplary embodiments of the present application are described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application. Therefore, all of these modifications are intended to be included within the scope of the present application as defined by the claims. It should be understood that the above is an explanation of the present application and should not be considered to be limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present application is defined by the claims and their equivalents.

Claims

1. An electronic device for training and processing a neural network model, comprising: A central processing unit, comprising a first memory corresponding to the central processing unit; as well as an artificial intelligence processing unit, comprising a second memory corresponding to the artificial intelligence processing unit, The central processing unit is configured to store and control intermediate tensors in the neural network model training process, including: Using the artificial intelligence processing unit to perform forward calculations on the neural network model; For a first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in the second memory; Determine whether a current storage usage rate of the second memory exceeds a first threshold, and if it is determined that the current storage usage rate exceeds the first threshold, transfer the first intermediate tensor to the first memory; Performing backward calculation on the neural network model using the artificial intelligence processing unit; and Before using the first intermediate tensor for calculation in the backward calculation, the storage location of the first intermediate tensor is determined. When it is determined that the storage location is the second memory, the first intermediate tensor is directly used by the artificial intelligence processing unit. When it is determined that the storage location is the first memory, the first intermediate tensor is retrieved from the first memory to the second memory for use in the backward calculation.

2. The electronic device according to claim 1, characterized in that: The process of generating the first intermediate tensor in the forward calculation corresponds to the first processing queue, the process of transferring the first intermediate tensor to the first memory corresponds to the second processing queue, the process of using the first intermediate tensor for calculation in the backward calculation corresponds to the third processing queue, and the process of retrieving the first intermediate tensor to the second memory corresponds to the fourth processing queue, The central processing unit performs storage control on the intermediate tensors in the neural network model training process, and further includes: Synchronously controlling the first processing queue and the second processing queue so that the second processing queue is executed after the first processing queue is finished; and The third processing queue and the fourth processing queue are synchronously controlled so that the third processing queue is executed after the fourth processing queue is executed.

3. The electronic device according to claim 1, characterized in that The central processing unit performs storage control on the intermediate tensors in the neural network model training process and further comprises: For a second intermediate tensor generated in the forward calculation, when determining to transfer the second intermediate tensor to the first memory, determining whether a current storage usage rate of the second memory exceeds a second threshold; and If it is determined that the second threshold is exceeded, a fifth processing queue for generating other intermediate tensors in the forward calculation and a sixth processing queue for transferring the second intermediate tensor to the first memory are synchronously controlled so as to suspend the fifth processing queue until the current storage usage rate does not exceed the second threshold, The other intermediate tensors are one or more intermediate tensors different from the second intermediate tensor.

4. The electronic device according to claim 1, characterized in that The central processing unit performs storage control on the intermediate tensors in the neural network model training process and further comprises: In the process of using the third intermediate tensor for calculation in the backward calculation, when determining to retrieve the third intermediate tensor from the first memory to the second memory, determining whether a current storage usage rate of the second memory exceeds a third threshold; and If it is determined that the third threshold is exceeded, synchronously controlling the seventh processing queue that uses other intermediate tensors in the backward calculation and the eighth processing queue that retrieves the third intermediate tensor from the first memory to the second memory, so that the eighth processing queue is suspended until the current storage usage rate does not exceed the third threshold, The other intermediate tensors are one or more intermediate tensors different from the third intermediate tensor.

5. The electronic device according to claim 1, characterized in that: There are multiple artificial intelligence processing units, each of which includes a second memory corresponding to it, and the central processing unit is configured to perform storage control of the intermediate tensor for each artificial intelligence processing unit respectively.

6. The electronic device according to claim 1, characterized in that: The first intermediate tensor is an activation value tensor generated in the forward calculation.

7. The electronic device according to claim 1, characterized in that: The number of the artificial intelligence processing units is multiple, each artificial intelligence processing unit includes a second memory corresponding thereto, the first intermediate tensor is an activation value tensor generated in the forward calculation, and the central processing unit performs storage control on the intermediate tensors in the neural network model training process, and further includes: Splitting a weight tensor to be calculated with the activation value tensor into N parts, and distributing the parts to N artificial intelligence processing units among the plurality of artificial intelligence processing units, so that the calculation process of the activation value tensor and the weight tensor is performed in parallel by the N artificial intelligence processing units; In the case where it is determined that the activation value tensor is transferred to the first memory, the activation value tensor is divided into N parts, and for the N artificial intelligence processing units, 1 / N parts of the activation value tensor are transferred to the first memory respectively; and Before using the activation value tensor in the backward calculation, 1 / N parts of the activation value tensor are retrieved respectively, and the complete activation value tensor is obtained by using a collective communication operation.

8. The electronic device according to claim 1, characterized in that: The central processing unit implements storage control of intermediate tensors in the neural network model training process as a first instruction code implemented by a context manager, and the first instruction code is executed by the central processing unit, wherein the neural network model is implemented as a second instruction code decoupled from the first instruction code.

9. A method for training a neural network model, characterized in that: The method is applicable to an electronic device, the electronic device comprising a central processing unit and an artificial intelligence processing unit, the central processing unit comprising a first memory corresponding to the central processing unit, the artificial intelligence processing unit comprising a second memory corresponding to the artificial intelligence processing unit, the method comprising: Perform forward calculations on the neural network model; For a first intermediate tensor generated in the forward calculation, storing the first intermediate tensor in the second memory; Determine whether a current storage usage rate of the second memory exceeds a first threshold, and if it is determined that the current storage usage rate exceeds the first threshold, transfer the first intermediate tensor to the first memory; Performing backward calculation on the neural network model; and Before using the first intermediate tensor for calculation in the backward calculation, the storage location of the first intermediate tensor is determined. When it is determined that the storage location is the second memory, the first intermediate tensor is directly used by the artificial intelligence processing unit. When it is determined that the storage location is the first memory, the first intermediate tensor is retrieved from the first memory to the second memory for use in the backward calculation.

10. The method according to claim 9, characterized in that The process of generating the first intermediate tensor in the forward calculation corresponds to the first processing queue, the process of transferring the first intermediate tensor to the first memory corresponds to the second processing queue, the process of using the first intermediate tensor for calculation in the backward calculation corresponds to the third processing queue, and the process of retrieving the first intermediate tensor to the second memory corresponds to the fourth processing queue, Wherein, the method further comprises: Synchronously controlling the first processing queue and the second processing queue so that the second processing queue is executed after the first processing queue is finished; and The third processing queue and the fourth processing queue are synchronously controlled so that the third processing queue is executed after the fourth processing queue is executed.

11. The method according to claim 9, characterized in that The method further comprises: For a second intermediate tensor generated in the forward calculation, when determining to transfer the second intermediate tensor to the first memory, determining whether a current storage usage rate of the second memory exceeds a second threshold; and If it is determined that the second threshold is exceeded, a fifth processing queue for generating other intermediate tensors in the forward calculation and a sixth processing queue for transferring the second intermediate tensor to the first memory are synchronously controlled so as to suspend the fifth processing queue until the current storage usage rate does not exceed the second threshold, The other intermediate tensors are one or more intermediate tensors different from the second intermediate tensor.

12. The method according to claim 9, characterized in that The method further comprises: In the process of using the third intermediate tensor for calculation in the backward calculation, when determining to retrieve the third intermediate tensor from the first memory to the second memory, determining whether a current storage usage rate of the second memory exceeds a third threshold; and If it is determined that the third threshold is exceeded, synchronously controlling the seventh processing queue that uses other intermediate tensors in the backward calculation and the eighth processing queue that retrieves the third intermediate tensor from the first memory to the second memory, so that the eighth processing queue is suspended until the current storage usage rate does not exceed the third threshold, The other intermediate tensors are one or more intermediate tensors different from the third intermediate tensor.

13. The method according to claim 9, characterized in that There are multiple artificial intelligence processing units, each of which includes a second memory corresponding to it, and storage control of intermediate tensors is performed separately for each artificial intelligence processing unit.

14. The method according to claim 13, characterized in that The first intermediate tensor is an activation value tensor generated in the forward calculation.

15. The method according to claim 9, characterized in that The number of the artificial intelligence processing units is multiple, each artificial intelligence processing unit includes a second memory corresponding thereto, the first intermediate tensor is an activation value tensor generated in the forward calculation, and the method further includes: Splitting a weight tensor to be calculated with the activation value tensor into N parts, and distributing the parts to N artificial intelligence processing units among the plurality of artificial intelligence processing units, so that the calculation process of the activation value tensor and the weight tensor is performed in parallel by the N artificial intelligence processing units; In the case where it is determined that the activation value tensor is transferred to the first memory, the activation value tensor is divided into N parts, and for the N artificial intelligence processing units, 1 / N parts of the activation value tensor are transferred to the first memory respectively; and Before using the activation value tensor in the backward calculation, 1 / N parts of the activation value tensor are retrieved respectively, and the complete activation value tensor is obtained by using a collective communication operation.

16. The method according to claim 9, characterized in that The method is implemented as a first instruction code implemented by a context manager, the first instruction code is executed by the central processing unit, wherein the neural network model is implemented as a second instruction code decoupled from the first instruction code.

17. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 9 to 16.

Images