An AI model generation method and device

Abstract

Provided are a method and device for generating an AI model, which relate to the technical field of artificial intelligence and can improve the efficiency of generating an AI model and reduce resource consumption to a certain extent. The method comprises: a first processor constructing a target optimization function with operation instruction information as an optimization variable based on the execution time length of each operator in a plurality of operators of an AI model, the memory occupancy size of a second memory of a second processor during operation of each operator, and the memory unloading time length; and globally optimizing the target optimization function to obtain operation instruction information for indicating whether each operator in the plurality of operators performs a target operation, the target operation comprising a recalculation operation or a memory unloading operation; and sending an adjustment instruction comprising the operation instruction information to the second processor to instruct the second processor to execute the plurality of operators according to the operation instruction information to train the AI model to obtain a target AI model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for generating an AI model. Background Technology

[0002] With the continuous improvement of AI chip computing power, the increase in communication bandwidth, and the continuous optimization of AI model training methods, training high-performance AI models on large-scale equipment clusters containing AI chips has become an important research hotspot in academia and industry.

[0003] Because AI models have complex structures (e.g., many network layers) and a large number of parameters, existing AI model training methods generally suffer from the problems of long training times and high resource consumption (such as computing power and storage resources). Summary of the Invention

[0004] This application provides a method and apparatus for generating AI models, which can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a method for generating an AI model, applied to a computing device including a first processor and a second processor. The memory of the first processor is called first memory, and the memory of the second processor is called second memory. The first processor controls the second processor to execute multiple operators of the AI ​​model to train the AI ​​model. The method includes: the first processor constructing a target optimization function with operation instruction information as the optimization variable based on the execution time of each operator in the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator; and performing global optimization on the target optimization function to obtain operation instruction information; the operation instruction information is used to indicate whether a target operation is executed for each operator in the multiple operators, the target operation including a recalculation operation or a memory unloading operation; and sending an adjustment instruction to the second processor, the adjustment instruction including the operation instruction information, the adjustment instruction being used to instruct the second processor to execute multiple operators according to the operation instruction information to train the AI ​​model to obtain a target AI model. The memory unloading time is the time it takes for the second processor to obtain the data required to execute the first operator from the first memory when data in the second memory is unloaded to the first memory, and the first operator is any one of the multiple operators.

[0007] In the AI ​​model generation method provided in this application embodiment, the first processor can model the optimization strategy (recomputation or memory unloading) based on the operator execution time, the memory usage of the second memory (the memory of the second processor) during the operator operation, and the operator memory unloading time. In this way, the optimal choice can be made between recomputation operation and memory unloading operation. That is, during the AI ​​model training process, the appropriate operation can be flexibly and dynamically selected according to the actual situation of the AI ​​model's operators. This overcomes the problems of high resource consumption and low training efficiency caused by using recomputation or memory unloading technology alone, and can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0008] In one possible implementation, the AI ​​model generation method provided in this application embodiment further includes: a first processor determining the execution time of each operator in the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of multiple operators during the training process of the AI ​​model.

[0009] The operator load information includes at least one of the following: the dependency relationship of multiple operators in the AI ​​model, the execution order of multiple operators, the memory access information of each operator in the multiple operators, the cache hit rate of each operator or the input and output size of each operator, and the memory usage of each operator.

[0010] In one possible implementation, the first processor determines the execution time of the first operator among the multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process. This includes: the first processor determining the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the computational load of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0011] In one possible implementation, the first processor determines the memory usage of the second memory during the operation of the first operator among the multiple operators of the AI ​​model based on the load information of multiple operators during the training process of the AI ​​model. This includes: the first processor determining the memory usage of the second memory during the operation of the first operator based on the dependency relationship of multiple operators, the execution order of multiple operators, and the input and output size of the first operator.

[0012] In one possible implementation, the load information of multiple operators in the AI ​​model also includes the memory usage of each operator in the multiple operators. The first processor determines the memory unloading time of the first operator in the multiple operators of the AI ​​model based on the load information of multiple operators during the training process of the AI ​​model. This includes: the first processor determining the memory unloading time of the first operator based on the memory usage of the first operator, the bandwidth of the second processor, the bandwidth utilization of the second processor, and the latency of the second processor executing the first operator.

[0013] The aforementioned operator execution time, the size of the second memory during operator operation, and the memory unloading time are indicators related to the hardware capabilities of the device and the characteristics of the operator. Furthermore, when modeling the optimization strategies for operator recomputation and memory unloading, these indicators were taken into consideration to avoid the problem of low efficiency and high resource consumption in the training process of AI models caused by selecting a certain optimization strategy based solely on the theoretical operator computation amount and memory usage.

[0014] Secondly, embodiments of this application provide a computing device, including a construction module, a computing module, and a sending module. The construction module is used to construct a target optimization function with operation instruction information as the optimization variable, based on the execution time of each operator in a plurality of operators of an AI model, the memory usage of a second memory during the operation of each operator, and the memory unloading time. The second memory is the memory of a second processor, and the memory unloading time is the time it takes for the second processor to obtain the data required to execute the first operator from the first memory when data in the second memory is unloaded to the first memory. The first operator is any one of the plurality of operators, and the first memory is the memory of the first processor. The computing module is used to perform global optimization on the target optimization function to obtain operation instruction information. This operation instruction information indicates whether to execute a target operation for each of the plurality of operators. The target operation includes a recalculation operation or a memory unloading operation. The sending module is used to send an adjustment instruction to the second processor. This adjustment instruction includes the operation instruction information and instructs the second processor to execute the plurality of operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

[0015] In one possible implementation, the computing device further includes a determining module, which is used to determine the execution time of each of the multiple operators in the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of multiple operators during the training of the AI ​​model.

[0016] The operator load information includes at least one of the following: the dependency relationship of multiple operators in the AI ​​model, the execution order of multiple operators, the memory access information of each operator in the multiple operators, the cache hit rate of each operator or the input and output size of each operator, and the memory usage of each operator.

[0017] In one possible implementation, the aforementioned determining module is specifically used to determine the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the computational load of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0018] In one possible implementation, the aforementioned determining module is specifically used to determine the memory usage of the second memory during the operation of the first operator based on the dependencies of multiple operators, the execution order of multiple operators, and the input and output size of the first operator.

[0019] In one possible implementation, the load information of multiple operators in the AI ​​model also includes the memory usage of each of the multiple operators; the aforementioned determining module is specifically used to determine the memory unloading time of the first operator based on the memory usage of the first operator, the bandwidth utilization of the second processor, the bandwidth of the second processor, and the latency of the second processor in executing the first operator.

[0020] In the first and second aspects mentioned above, the objective optimization function takes the execution time of the operator and the memory unloading time as the optimization objectives, and takes the memory usage of the second memory during the operation of the operator as the constraint.

[0021] In one possible implementation, the AI ​​model includes n operators, where n is an integer greater than or equal to 2; the objective optimization function is:

[0022]

[0023] Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i E represents the memory usage of the second memory during the execution of the i-th operator. total E represents the total memory usage for training the AI ​​model. max Indicates the constraint value of available memory; x i Used to indicate whether to perform the target operation on the i-th operator, where when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0024] The optimization objective of the above objective function is The optimization objective is to minimize the additional time overhead caused by recalculating operators or unloading memory during the training of the AI ​​model, thus ensuring optimal performance. The constraints of the objective function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the actual memory size occupied by training the AI ​​model shall not exceed the size of the available memory constrained by the AI ​​chip.

[0025] In the first and second aspects mentioned above, the objective optimization function takes the memory usage of the second memory during the operator's operation as the optimization objective and the execution time of the operator and the memory unloading time as constraints.

[0026] In one possible implementation, the AI ​​model includes n operators, where n is an integer greater than or equal to 2; the objective optimization function is:

[0027]

[0028] Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second memory during the execution of the i-th operator. target This represents the total training time for the AI ​​model; x i Used to indicate whether to perform the target operation on the i-th operator, where when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0029] The optimization objective of the above objective function is The meaning of this optimization objective is to maximize the memory saved by recalculating operators or unloading memory during the training process of the AI ​​model, that is, to minimize the memory consumed during the training process of the AI ​​model.

[0030] The constraints of the above objective function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent training the AI ​​model does not exceed the estimated total time. The total time spent training the AI ​​model is the sum of the total execution time of all operators and the additional time consumed due to recomputation operations or memory unloading operations.

[0031] In the first and second aspects mentioned above, the objective optimization function takes the execution time of the operator, the memory unloading time, and the memory usage of the second memory during the operation of the operator as optimization objectives, and takes the execution time of the operator, the memory unloading time, and the memory usage of the second memory during the operation of the operator as constraints.

[0032] In one possible implementation, the AI ​​model includes n operators, where n is an integer greater than or equal to 2; the objective optimization function is:

[0033]

[0034] Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second memory during the execution of the i-th operator. target E represents the total time spent training the AI ​​model. total E represents the total memory usage for training the AI ​​model. max Represents the constraint value for available memory; α and β are weighting coefficients; x i Used to indicate whether to perform the target operation on the i-th operator, where when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0035] The optimization objective of the above objective function is The optimization objective means that during the training of the AI ​​model, the memory saved by recalculating operators or unloading memory should be maximized, while the additional time overhead caused by recalculating operators or unloading memory should be minimized. In other words, the memory consumed during the training of the AI ​​model should be minimized and the performance optimized.

[0036] The constraints of the above objective function are: and The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent training the AI ​​model shall not exceed the estimated total time, and the actual memory size occupied by the AI ​​model during training shall not exceed the available memory size constrained by the AI ​​chip.

[0037] In one possible implementation, when x i =1, and min(T) i ,T swap_i ) = T i When x ∈ [0, 1], perform a recalculation operation on the i-th operator; when .... i =1, and min(T)i ,T swap_i ) = T swap_i When the i-th operator is in the i-th position, a memory unloading operation is performed.

[0038] Thirdly, embodiments of this application provide a computing device, including a memory and at least one processor connected to the memory. The at least one processor includes a first processor and a second processor. The memory is used to store computer program code, which includes computer instructions. When the computer instructions are executed by the at least one processor, the first processor performs the method described in any one of the first aspects and its possible implementations, and the second processor performs multiple operators of an AI model to train the AI ​​model.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed on a computer, perform any one of the methods of the first aspect and its possible implementations.

[0040] Fifthly, embodiments of this application provide a computer program product comprising computer instructions that, when executed on a computer, perform any one of the methods of the first aspect and its possible implementations.

[0041] In a sixth aspect, embodiments of this application provide a chip system, including: a processor, configured to call and run a computer program from a memory, causing a terminal device equipped with the chip system to perform the method of the first aspect and any of its possible implementations.

[0042] It should be understood that the beneficial effects of the technical solutions of the second to sixth aspects of this application and the corresponding possible implementations can be referred to the above-described technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0043] Figure 1 A schematic diagram of the structure of a Transformer model provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application;

[0045] Figure 3 One of the schematic diagrams of an AI model generation method provided in this application embodiment;

[0046] Figure 4 A second schematic diagram illustrating a method for generating an AI model provided in an embodiment of this application;

[0047] Figure 5This is one of the structural schematic diagrams of a computing device provided in an embodiment of this application;

[0048] Figure 6 This is a second schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0049] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0050] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, rather than to describe a specific order of objects. For example, "first processor" and "second processor," etc., are used to distinguish different processors, rather than to describe a specific order of processors.

[0051] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0052] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, and "multiple" can also be described as at least two.

[0053] First, some concepts involved in the embodiments of this application will be explained.

[0054] 1. AI Model

[0055] In the field of artificial intelligence (AI) technology, a mathematical model is constructed to achieve a specific function (such as object detection, image processing, natural language processing, etc.). The process of applying an AI model to achieve a specific function includes model training and model inference. Model training results in a high-performance AI model, while model inference uses the trained AI model to implement the specific function.

[0056] AI models consist of model architecture and model parameters. Typically, after the AI ​​model architecture is designed, it can be continuously trained using machine learning methods (such as deep learning) based on a large number of training samples (the training process can be understood as continuously updating the AI ​​model's parameters) to obtain an AI model capable of performing specific functions. The AI ​​model training process includes: inputting training samples into the current AI model, obtaining prediction results through the AI ​​model's processing (i.e., mathematical operations); then comparing the prediction results with the actual results of the training samples to determine the model loss; and then updating the AI ​​model's parameters based on the model loss. By repeatedly executing the above process, a higher-performing AI model is obtained.

[0057] Optionally, commonly used AI models may include, but are not limited to, the following models: convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM) network, and Transformer network.

[0058] Among them, CNN is a deep learning model used for image recognition and classification. RNN is a recurrent neural network used to process sequential data (such as text and speech). It maintains historical information through recurrent layers to understand long-term dependencies in the data. LSTM is a variant of RNN. The special feature of LSTM is that it has an internal memory unit that can store long-term information, enabling it to handle long-term dependency problems and possess long-term memory capabilities, remembering long-term dependencies in the data. Transformer is a neural network model based on an attention mechanism. It captures the relationships between data through attention and can be used for processing sequential data (such as speech recognition, machine translation, and text classification).

[0059] Understandably, AI models perform a large number of complex mathematical operations. In other words, an AI model comprises numerous operators (operators can be understood as mappings from one function space to another). These operators are executed according to certain dependencies and in a specific order to achieve the AI ​​model's functionality. Among the many operators in an AI model, some have low computational complexity, while others have high computational complexity.

[0060] For example, taking the Transformer mentioned above as an example, Figure 1 A schematic diagram of a Transformer structure is shown. (For example...) Figure 1As shown, a Transformer consists of multiple Transformer blocks connected in series, meaning the output of one Transformer block serves as the input to the next. The structure of each Transformer block can be found in [reference needed]. Figure 1 The linear module transforms the self-attention input (the input information of the Transformer) linearly to obtain three matrices: Q (Query), K (Key), and V (Value). Then, a dot product operation is performed on Q and K (dot product). Softmax normalizes the result of the dot product of Q and K. Dropout randomly discards the normalized result to improve the robustness of the Transformer. Finally, after linear processing by the self-attention output module and processing by the feed-forward network (FFN), the output of a Transformer block is obtained.

[0061] Combination Figure 1 The Linear, matrix multiplication, and FFN operations in the Transformer block mentioned above are matrix operators, while the remaining operations such as Softmax and Dropout are vector operators. Understandably, vector operators generally have lower computational complexity than matrix operators, and their execution time is usually shorter. However, in some cases, due to hardware limitations, the computational complexity of vector operators may exceed that of matrix operators, resulting in longer execution times for vector operators. For example, as the sequence length of a vector increases, the computational complexity of Softmax (a vector operator) in the Transformer increases quadratically. The time cost of Softmax operations on very long sequences can be extremely high, potentially exceeding the time cost of FFN operations.

[0062] 2. AI Chip

[0063] AI chips are modules specifically designed to handle large amounts of computational tasks in the field of artificial intelligence. AI chips can also be called AI accelerators, AI wafers, computing cards, or heterogeneous acceleration units, etc. AI accelerators are also a type of processor.

[0064] For example, AI chips may include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.

[0065] A GPU, also known as a visual processor or display chip, is a microprocessor specifically designed for performing image and graphics-related computations in personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). A GPU contains thousands of cores and can process thousands of threads simultaneously, enabling it to run massive amounts of computation in parallel.

[0066] FPGA stands for Programmable Logic Processor. Users can repeatedly program it according to their own needs. FPGA excels at handling short, intensive data access. FPGA allows users to program the number of circuits through tiny logic blocks and can handle any type of digital function.

[0067] An ASIC is a special-specification integrated circuit designed for a specific purpose, manufactured to meet the requirements of a particular user and the needs of a specific electronic system. Examples include tensor processing units (TPUs) and neural network processing units (NPUs). TPUs and NPUs are chips designed for machine learning, suitable for processing massive amounts of data, and possess powerful computing capabilities.

[0068] Based on the above conceptual description, currently, when training AI models using AI chips (such as GPUs), the complex structure of AI models (e.g., many network layers) and the large number of parameters pose certain challenges to the computing power and storage of AI chips.

[0069] Currently, one method to optimize the training process of AI models is to recompute all or some of the operators in the AI ​​model. Recompiling all operators is called full recomputed, while recompiling some operators is called selective recomputed. Essentially, recomputed computing is a technique that divides the computation process into multiple smaller chunks. Each chunk uses only a portion of the data for computation, caching some data (such as the AI ​​model's output) and discarding others (such as intermediate computation results). In subsequent computations, only the uncached data needs to be recomputed, and then the cached data is merged with the recomputed data. Recomputed computing can reduce the memory footprint of AI chips (such as GPU memory), allowing for the training of larger AI models with less memory. Furthermore, in some cases, it can also increase computational throughput.

[0070] Full recalculation refers to caching only some of the model's inputs and outputs, while leaving the data (or activation variables) from intermediate calculations uncached. During subsequent backpropagation (updating model parameters through backpropagation), all uncached data needs to be recalculated, resulting in additional computational overhead (approximately 30%-40% more computing power). For example, regarding the aforementioned... Figure 1 For the language class model of the Transformer architecture shown, full recalculation is used, the data output by each Transformer block is cached, and the activation variables in the intermediate operation process of the Transformer block are discarded.

[0071] Selective recomputation refers to caching some inputs and outputs of a model, caching a portion of intermediate computation data, and discarding some intermediate computation data. In other words, selective recomputation only repeats computation on a subset of data. Specifically, based on the theoretical computational load, selective recomputation can be performed on a subset of data with low computational complexity but high memory consumption on the AI ​​chip. This reduces the memory usage of the AI ​​model during training, improving training speed and effectiveness while ensuring that the memory consumption of the AI ​​chip remains relatively low. For example, regarding the aforementioned... Figure 1 For the large language model of the Transformer architecture shown, Softmax and Dropout operations generally consume a lot of memory, but their computational complexity is low. Therefore, Softmax and Dropout can be set as recomputable objects, while FFN is not recomputable.

[0072] Another method to optimize the AI ​​model training process is parameter offload technology. Parameter offload is essentially memory swapping, which involves storing the model's intermediate variables and training parameters in the CPU's memory instead of the AI ​​chip's memory, thereby reducing the AI ​​chip's memory footprint. During AI model training, the intermediate variables and training parameters are divided into multiple blocks, each assigned to a different AI chip for computation. When one AI chip needs to access intermediate variables and training parameters from other AI chips, it can retrieve them from the CPU's memory via the network, rather than from the memory of other AI chips. Memory offload reduces the AI ​​chip's memory footprint, allowing for the training of larger models based on the limited memory of the AI ​​chip. However, the bandwidth from the CPU to the AI ​​chip (i.e., host-to-device, H2D) is relatively small, and the speed at which the AI ​​chip retrieves intermediate variables and training parameters from the CPU is slow and time-consuming. Therefore, while the parameter offload technology achieves the memory gain of the AI ​​chip, it also causes a significant performance degradation. Understandably, the performance mentioned above mainly refers to the time cost or speed of training the AI ​​model, and time cost or speed factors affect the training efficiency of the AI ​​model.

[0073] It should be noted that since parameter unloading is essentially memory unloading, for ease of understanding and description, parameter exchange will be described as memory unloading in the following embodiments.

[0074] In summary, existing methods for training AI models suffer from time consumption and high resource consumption (such as computing and storage resources). To address these issues, this application provides a method for generating an AI model, applied to a computing device including a first processor and a second processor. The second processor executes multiple operators of the AI ​​model to train it, while the first processor performs the following process: Based on the execution time of each operator in the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator, the first processor constructs a target optimization function with operation instruction information as the optimization variable; and performs global optimization on the target optimization function to obtain operation instruction information, which indicates whether a target operation should be performed on each of the multiple operators. The target operation includes a recalculation operation or a memory unloading operation; further, the first processor sends an adjustment instruction to the second processor, which includes the operation instruction information. This adjustment instruction instructs the second processor to execute multiple operators according to the operation instruction information to train the AI ​​model and obtain the target AI model. Among them, for any one of the multiple operators (called the first operator), the memory unloading time is the time it takes for the second processor to obtain the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory. The first memory is the memory of the first processor, and the second memory is the memory of the second processor.

[0075] The AI ​​model generation method provided in this application can select the optimal operation between recomputation and memory unloading. That is, during the AI ​​model training process, the appropriate operation can be flexibly and dynamically selected according to the actual situation of the AI ​​model's operators. This overcomes the problems of high resource consumption and low training efficiency caused by using recomputation or memory unloading technology alone. It can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0076] Optionally, the first processor in the aforementioned computing device can be a general-purpose central processing unit (CPU) or other general-purpose processor, while the second processor can be an AI chip, such as a GPU, NPU, or TPU. The CPU is primarily responsible for the logical calculation and logical control functions of the workload, while the AI ​​chip is primarily responsible for a large number of complex calculation functions.

[0077] For example, Figure 2 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application. Figure 2 The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. For example... Figure 2As shown, the computing device may include: a processor 201, a memory 202, a communication interface 203, and an AI accelerator 205. The processor 201, memory 202, communication interface 203, and AI accelerator 205 can be connected to each other via a bus 204 or other means.

[0078] The processor 201 is the control center of the computing device. The processor 201 can be a CPU or other general-purpose processors, such as microprocessors or any conventional processors.

[0079] The controller in processor 201 is the nerve center and command center of the computing device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Optionally, processor 201 may also include a memory for storing instructions and data. For example, processor 201 may include one or more CPUs, such as... Figure 2 CPU 0 and CPU 1 are shown in the diagram.

[0080] The memory 202 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical memory, disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. In this embodiment, the memory 202 may store computer instructions and other information.

[0081] In one possible implementation, the memory 202 may exist independently of the processor 201. The memory 202 can be connected to the processor 201 via a bus 204 and is used to store data, instructions, or program code. When the processor 201 calls and executes the instructions or program code stored in the memory 202, it can implement the relevant steps in the method provided in the embodiments of this application.

[0082] In another possible implementation, the memory 202 can also be integrated with the processor 201.

[0083] The communication interface 203 can be a transceiver module used for communication with other devices or communication networks, such as Ethernet, RAN, and wireless local area networks (WLAN). The communication interface 203 can receive commands, messages, or data. The transceiver module can be a transceiver or similar device. Optionally, the communication interface 203 can also be a transceiver circuit located within the processor 201, used to implement the processor's signal input and signal output. The communication interface 203 can be a wired interface (port), such as a fiber distributed data interface (FDDI) or a gigabit Ethernet (GE) interface; alternatively, the communication interface 203 can also be a wireless interface.

[0084] AI accelerator 205 is a processor heterogeneous to the aforementioned processor 201. AI accelerator 205 is a module specifically designed to handle large-scale computational tasks in the field of artificial intelligence. AI accelerator 205 is also connected to other components via bus 204. AI accelerator 205 may include GPUs, FPGAs, ASICs, etc. For a more detailed description of AI accelerator 205, please refer to the conceptual introduction section of the above embodiments; it will not be repeated here.

[0085] The aforementioned processor 202 (i.e., the central processing unit CPU) is a general-purpose computing module, primarily responsible for the logical calculation and control functions of the workload. It can efficiently handle tasks with a single complex calculation sequence, but its performance in large-scale calculations is relatively low. Therefore, the processor can distribute large-scale calculation tasks to the AI ​​accelerator 205. After completing the calculation, the AI ​​accelerator 205 returns the calculation result to the processor 201.

[0086] Bus 204 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. The bus can also be divided into serial bus and parallel bus. For ease of representation, Figure 2 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0087] It should be noted that, Figure 2 The computing device shown is merely one example of a computing device; it can have more than Figure 2The more or fewer components shown can be combined into two or more components, or they can have different component configurations.

[0088] Based on the above, this application provides a method for generating an AI model. This method can be executed by a first processor (such as a CPU) in a computing device. The first processor executes the scheme to generate an optimization strategy for adjusting the AI ​​model training process, and controls a second processor to execute multiple operators of the AI ​​model using the optimization strategy to train the AI ​​model, thereby generating a target AI model. The memory of the first processor is referred to as first memory, and the memory of the second processor is referred to as second memory. Figure 3 As shown, the AI ​​model generation method provided in this application embodiment includes S301-S303.

[0089] S301, the first processor constructs a target optimization function with operation instruction information as the optimization variable based on the execution time of each operator in the multiple operators of the AI ​​model, the memory occupation size of the second memory during the operation of each operator, and the memory unloading time.

[0090] It should be understood that an AI model comprises multiple operators, and running an AI model is the process of executing these multiple operators according to their call relationships and execution order. The execution time of each operator, the amount of secondary memory occupied during the execution of each operator, and the memory unloading time are obtained based on the iteration results of any one of the training processes (i.e., multiple iterations) in which the second processor executes the AI ​​model's training process multiple times.

[0091] For any operator in the AI ​​model (hereinafter referred to as the first operator), the execution time of the first operator is the time required for the second processor to execute the first operator (i.e., the time required for the forward computation of the first operator). The execution time of the first operator can also be referred to as the operator performance of the first operator. The memory usage of the second memory during the execution of the first operator is the size of the storage space currently occupied by the second memory during the execution of the first operator by the second processor. The memory unloading time is the time required for the second processor to retrieve the data required for executing the first operator from the first memory when the data in the second memory is unloaded to the first memory. The memory unloading time of the first operator can also be referred to as the memory unloading cost of the first operator.

[0092] Optionally, the execution order of some operators in the AI ​​model may be sequential, while the execution order of others may be parallel. Referring to the example, it should be noted that in the case of sequential execution, when the first operator is currently executing, the memory usage of the second memory during the execution of the first operator is the memory required to run the first operator. In the case of parallel execution, when the first operator is executing, the second operator, which is executing in parallel with it, is also running. Therefore, the memory usage of the second memory during the execution of the first operator is the sum of the memory required to run the first operator and the memory required to run the second operator.

[0093] Understandably, if the first operator is recalculated, the time required to recalculate the first operator during the reverse calculation process is the execution time of the first operator, that is, the extra time required to recalculate the first operator.

[0094] The memory footprint of the second memory during the execution of the first operator is determined by the amount of storage space occupied in the second memory during the normal execution of the first operator by the second processor (where the second processor and its memory participate in the operator calculation, without the application of memory unloading technology). It's understandable that if the first operator is recalculated, its related data can be discarded during the forward calculation, thus saving the memory occupied by the first operator. In other words, the memory footprint of the second memory represents the amount of memory saved by recalculating the first operator. The memory unloading time is the additional time consumed during the execution of the first operator when memory unloading technology is used.

[0095] Based on the execution time of each operator in the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator, a comprehensive determination is made as to whether the execution process of each operator needs optimization, i.e., whether to perform a recalculation operation or a memory unloading operation on the first operator; and it is further determined which operation to use for optimization of the first operator, i.e., whether to use a recalculation operation or a memory unloading operation. In this embodiment, to achieve this objective, an optimization function is constructed based on the execution time, memory usage, and memory unloading time. The optimization variable of this optimization function is operation indication information used to optimize the execution process of each operator. This operation indication information is used to indicate whether to perform a target operation on each of the multiple operators in the AI ​​model, and the target operation includes a recalculation operation or a memory unloading operation.

[0096] Optionally, the target optimization function constructed based on the operator's execution time, the memory usage of the second memory during the operator's operation, and the memory unloading time can include the following three functions.

[0097] The first type: The objective optimization function takes the execution time of the operator and the memory unloading time as the optimization objectives, and takes the memory usage of the second memory during the operation of the operator as the constraint.

[0098] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the objective optimization function can be the function shown in the following formula (1):

[0099]

[0100] In formula (1), T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i E represents the memory usage of the second memory during the execution of the i-th operator. total E represents the total memory usage for training the AI ​​model. max This represents the constraint value for available memory. E total This is the theoretically estimated total memory usage, E max This is the minimum amount of memory available during the training of an AI model. It should be understood that E... max It is related to the hardware parameters of the AI ​​chip; for different AI chips, E max They may be different.

[0101] x in formula (1) i Used to indicate whether to perform the target operation on the i-th operator, where when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0102] Combining the above formula (1), the optimization objective of the objective function is... The optimization objective is to minimize the additional time overhead caused by recalculating operators or unloading memory during the training of the AI ​​model, thus ensuring optimal performance. The constraints of the objective function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the actual memory size occupied by training the AI ​​model shall not exceed the size of the available memory constrained by the AI ​​chip.

[0103] The second type: The objective optimization function takes the memory usage of the second memory during the operator's operation as the optimization objective, and the execution time of the operator and the memory unloading time as constraints.

[0104] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the objective optimization function can be the function shown in the following formula (2):

[0105]

[0106] In formula (2), T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second inner layer during the execution of the i-th operator. target This represents the total time required to train the AI ​​model (which is the theoretically estimated total time).

[0107] Similarly, x in formula (2) i Used to indicate whether to perform the target operation on the i-th operator, when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0108] Combining the above formula (2), the optimization objective of the objective function is... The optimization objective is to maximize the memory savings achieved through operator recalculation or memory unloading operations during AI model training, thereby minimizing the memory consumption during AI model training. The constraints of the objective function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent training the AI ​​model does not exceed the estimated total time. The total time spent training the AI ​​model is the sum of the total execution time of all operators and the additional time consumed due to recomputation operations or memory unloading operations.

[0109] The third type: The objective optimization function takes the execution time of the operator, the memory unloading time, and the memory usage as the optimization objectives, and takes the execution time of the operator, the memory unloading time, and the memory usage as the constraints.

[0110] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the objective optimization function can be the function shown in formula (3):

[0111]

[0112] In formula (3), T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second memory during the execution of the i-th operator. target E represents the total time spent training the AI ​​model. total E represents the total memory usage for training the AI ​​model.max This represents the constraint value of available memory; α and β are weighting coefficients, which are used to unify the dimensions of time and memory, and to balance the importance of these two quantities in the optimization objective.

[0113] Similarly, x in formula (3) i Used to indicate whether to perform the target operation on the i-th operator, when x i When x = 1, perform the target operation on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

[0114] Combining the above formula (3), the optimization objective of the objective function is... The optimization objective is to maximize the memory savings from recalculating operators or unloading memory during AI model training, while minimizing the additional time overhead caused by these operations. In other words, it aims to minimize memory consumption and optimize performance during AI model training. The constraints of the objective function are: and The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent training the AI ​​model shall not exceed the estimated total time, and the actual memory size occupied by the AI ​​model during training shall not exceed the available memory size constrained by the AI ​​chip.

[0115] S302. Perform global optimization on the objective optimization function to obtain operation indication information for indicating whether to perform the objective operation on each of the multiple operators. The objective operation includes recalculation operation or memory unloading operation.

[0116] It should be noted that when a target operation needs to be performed on the i-th operator, the target operation performed on the i-th operator is the operation with the shortest time consumption between recomputation and memory unloading operations. That is, when x i =1, and min(T) i T swap_i ) = T i When x ∈ [0, 1], perform a recalculation operation on the i-th operator; when .... i =1, and min(T) i ,T swap_i ) = T swap_i At that time, the memory unloading operation is performed on the i-th operator.

[0117] The objective function described above can be solved using an optimization algorithm. In this embodiment, the performance gains and losses caused by memory unloading and recomputation are considered together, and an optimization algorithm is used to perform global optimization on the objective function to obtain the global optimal solution.

[0118] For the first type of objective optimization function shown in the above formula (1), the objective optimization function can be transformed into a 0-1 knapsack problem, and the objective optimization function can be solved by dynamic programming algorithm.

[0119] Specifically, the transformation process of formula (1) is as follows:

[0120]

[0121] It should be understood that the prototype of the 0-1 knapsack problem is: there are n items and a knapsack with a maximum capacity (the weight of the items) of w, where the weight of any one item is w. i The value is v i The problem asks for the optimal number of items to put into a knapsack to maximize the total value of the items. For each item, its state can be either included in the knapsack or not included; that is, the state of an item can be either 0 or 1, where 0 represents not included and 1 represents included. Therefore, the state of item i can be denoted as x. i x i ∈(0,1).

[0122] The objective function for the knapsack problem described above is:

[0123]

[0124] The 0-1 knapsack problem can be solved using dynamic programming. The principle of dynamic programming is to transform a multi-stage decision-making process into a series of single-stage decision problems. By utilizing the recursive relationship between each stage, the optimal decision for each stage is determined one by one, and finally the optimal decision result of the multi-stage decision is stacked. For a detailed solution process of the 0-1 knapsack problem using dynamic programming, please refer to the existing technical documents. This application will not elaborate on the embodiments.

[0125] Referring to the description of the knapsack problem, in this embodiment, whether each operator performs the target operation is equivalent to whether each item is put into the knapsack in the knapsack problem. Therefore, the execution time, memory usage, and memory unloading time of the operators are modeled as a 0-1 knapsack problem, and x is obtained by solving it through dynamic programming. i This allows us to determine whether each operator needs to perform the target operation, and which operation to perform (recompute or unload memory).

[0126] For the second type of objective optimization function shown in the above formula (2), the objective optimization function is in the form of a standard 0-1 knapsack problem, so the dynamic programming algorithm can be used directly to solve the objective optimization function.

[0127] For the third objective optimization function shown in formula (3), the memory usage and duration are unified by the coefficients α and β in formula (3), which transforms the objective problem into a single objective problem. It can also be transformed into a knapsack problem for solution. The specific algorithm reasoning process can be found in existing technical materials, which will not be detailed here.

[0128] S303, the first processor sends an adjustment instruction to the second processor, the adjustment instruction including operation instruction information.

[0129] The adjustment instruction is used to instruct the second processor to execute multiple operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

[0130] In this embodiment, the operation instruction information obtained by the first processor from solving the objective optimization function is an optimization strategy for optimizing the training process of the AI ​​model. The first processor can send adjustment instructions containing the operation instruction information to the second processor, so that the second processor can perform corresponding operations on one or more operators according to the operation instruction information. For example, if the operation instruction information indicates that the objective operation of the i-th operator is a recalculation operation, then when the second processor executes the i-th operator, it performs a recalculation operation on the i-th operator.

[0131] In one implementation, the first processor can monitor the current task execution status of the AI ​​model (including the execution status of each operator) and issue adjustment instructions based on the operator execution status. Specifically, the computing device may include a monitoring module and an instruction issuing module for monitoring the current task execution status. The monitoring module is responsible for monitoring the current task execution status and (e.g., through TaskScheduler) feeding back the operator execution status to the instruction issuing module. Based on the operator execution status fed back by the monitoring module, the instruction issuing module sends adjustment instructions to the second processor at a specific time point. Optionally, the monitoring module and the instruction issuing module can communicate with the instruction issuing module through inter-process communication (IPC). The monitoring module and the instruction issuing module can also be hardened hardware modules that communicate through a hardware interface.

[0132] In summary, the AI ​​model generation method provided in this application embodiment can model the optimization strategy (recomputation or memory unloading) based on the operator execution time, the memory usage of the second memory during operator operation, and the memory unloading time. In this way, the optimal choice can be made between recomputation and memory unloading operations. That is, during the AI ​​model training process, the appropriate operation can be flexibly and dynamically selected according to the actual situation of the AI ​​model's operators. This overcomes the problems of high resource consumption and low training efficiency caused by using recomputation or memory unloading techniques alone, and can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0133] Optionally, combined Figure 3 ,like Figure 4 As shown, prior to S301 above, the AI ​​model generation method provided in this application embodiment further includes S300.

[0134] S300, the first processor determines the execution time of each operator in the multiple operators of the AI ​​model, the memory usage of the second memory during the execution of each operator, and the memory unloading time of each operator based on the load information of multiple operators during the AI ​​model training process.

[0135] Optionally, the load information (also known as operator load characteristics) of multiple operators during the AI ​​model training process is obtained from the iteration results of any one of the training processes (i.e., multiple iterations) executed multiple times by the second processor during the AI ​​model training process.

[0136] The operator load information includes at least one of the following: the dependency relationship of multiple operators in the AI ​​model, the execution order of multiple operators, the memory access information of each operator in the multiple operators, the cache hit rate of each operator, the input and output size of each operator, or the memory usage of each operator.

[0137] In this context, the dependency relationship among multiple operators refers to the dependency relationship between the inputs and outputs of multiple operators. For example, if the input of one operator is the input of another operator, then the two operators have a dependency relationship.

[0138] The execution order of operators can be either parallel or serial.

[0139] Operator memory access information refers to the amount of data accessed by different levels of cache units.

[0140] The cache hit rate of an operator is the ratio of the number of hits to the total number of accesses (the sum of hits and misses) when reading data from the cache during the execution of the operator.

[0141] The input and output size of an operator includes the input size and the output size of the operator.

[0142] The memory footprint of an operator is the amount of memory occupied by the second processor used to execute that operator.

[0143] In one implementation, the first processor determines the execution time of the first operator among the multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, specifically including S300a.

[0144] S300a, the first processor determines the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the computational load of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0145] In one implementation, the process of determining the execution time of the first operator may include the following steps:

[0146] Step 1: Based on the memory access information and cache hit rate of the first operator, determine the data transfer volume, bandwidth utilization, and instruction execution efficiency during the execution of the first operator.

[0147] The data transfer volume of the first operator is the ratio of the memory access information of the first operator to the cache hit rate of the first operator.

[0148] Step 2: Determine the computation time of the first operator based on the computational load of the first operator, the computing power of the second processor, and the instruction execution efficiency of the first operator; determine the data transfer time of the first operator based on the data transfer volume of the first operator, the bandwidth utilization rate, and the bandwidth of the second processor.

[0149] For ease of understanding, the computational cost of the first operator is F. n The data transfer volume of the first operator is denoted as M. n The bandwidth utilization of the first operator is denoted as α. n The instruction execution efficiency of the first operator is denoted as β. n The computing power of the second processor is denoted as P. n The bandwidth of the second processor is denoted as B. n Then the data transfer time t of the first operator compute and t move The calculation method is as follows.

[0150]

[0151] Step 3: Select the larger of the computation time and data transfer time of the first operator as the execution duration of the first operator.

[0152] In one implementation, the first processor determines the memory usage of the first operator among the multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, specifically including S300b.

[0153] S300b: The first processor determines the memory usage of the second memory during the operation of the first operator based on the dependencies of multiple operators, the execution order of multiple operators, and the input and output sizes of the first operator.

[0154] In this embodiment of the application, the input and output size of the first operator is the memory size occupied by the first operator. Based on the dependency relationship and execution order of multiple operators, the current memory size occupied by the second memory during the operation of the first operator can be determined.

[0155] For example, consider two operators out of multiple operators, such as the first operator and the second operator. If the first operator and the second operator have a dependency relationship, the execution order is sequential (e.g., the first operator is executed first, then the second operator). If the first operator is currently executing, the memory usage in the second memory during the execution of the first operator is the same as the memory required to run the first operator. If the first operator and the second operator have no dependency relationship, the execution order is parallel (i.e., the second operator is running while the first operator is running). In this case, the memory usage in the second memory during the execution of the first operator is the sum of the memory required to run the first operator and the memory required to run the second operator.

[0156] In one implementation, the first processor determines the memory unloading time of the first operator among the multiple operators of the AI ​​model based on the load information of multiple operators during the training process of the AI ​​model, specifically including S300c.

[0157] S300c: The first processor determines the memory unloading time of the first operator based on the memory usage of the first operator, the bandwidth of the second processor, the bandwidth utilization of the second processor, and the latency of the second processor executing the first operator.

[0158] Alternatively, the memory unloading time of the first operator can be calculated as follows:

[0159]

[0160] Among them, T swap The memory unloading time of the first operator is represented by R, M represents the memory usage of the first operator, N represents the bandwidth of the second processing, and R represents the memory unloading time of the first operator. N T represents the bandwidth utilization of the second processing step. delay This indicates the delay in executing the first operator.

[0161] Optionally, in the embodiments of this application, the execution time of the operator and the memory usage of the second memory during the operation of the operator can also be obtained from the compiled graph or profiling. The compiled graph or profiling is an information collection tool in the training process of AI model.

[0162] It should be understood that obtaining profiling information requires enabling (or logging into) the profiling tool, which takes time. Therefore, in this embodiment, an offline knowledge base can be created, storing profiling information for AI models. The first processor can query the knowledge base. If the profiling information for the operator is found (e.g., by querying by operator name), the processor reads the operator execution time and the memory usage of the second memory during the operator's operation from the knowledge base. If the operator's profiling information is not found in the knowledge base, the processor can model the operator execution time and the memory usage of the second memory during the operator's operation according to the methods described in S300a and S300b.

[0163] Alternatively, if the profiling information for the operator is not found in the knowledge base, a profiling tool can be enabled to obtain the operator execution time and the amount of memory used in the second memory during the operator's operation from the latest profiling information online.

[0164] Combining S300a, S300b, and S300c, the operator execution time, the memory usage of the second memory during operator operation, and the operator memory unloading time are indicators related to the hardware capabilities of the device and the characteristics of the operator. Furthermore, when modeling the optimization strategies for operator recomputation and memory unloading, these indicators are taken into consideration, avoiding the problem of low efficiency and high resource consumption in the AI ​​model training process caused by selecting a certain optimization strategy solely based on the theoretical operator computation amount and memory usage.

[0165] It is understood that, in order to achieve the aforementioned functions, the computing device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0166] This application embodiment can divide the above-described computing device into functional modules based on the method example described above. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0167] When dividing each function into modules according to its corresponding function. Figure 5 A possible structural diagram of the computing device involved in the above embodiments is shown. The computing device includes a construction module 501, a computing module 502, and a transmission module 503.

[0168] The construction module 501 is used to execute S301 in the above method embodiment; the calculation module 502 is used to execute S302 in the above method embodiment; and the sending module 503 is used to execute S303 in the above method embodiment.

[0169] Optionally, the computing device provided in this application embodiment further includes a determination module 504, which is used to execute S300 (including S300a, S300b, and S300c) in the above method embodiment.

[0170] When using integrated units, Figure 6 A schematic diagram of another possible structure of the computing device involved in the above embodiments is shown. The computing device may include a processing module 601 and a communication module 602. The processing module 601 can be used to control and manage the operation of the computing device. For example, the processing module 601 can be used to support the computing device in executing S300, S301, and S302 in the above method embodiments, and / or other processes used in the technology described herein. The communication module 602 can be used to support communication between the computing device and other network entities. For example, the communication module 602 can be used to support the computing device in executing S303 in the above method embodiments. Optionally, as... Figure 6 As shown, the computing device may also include a storage module 603 for storing the program code and data of the computing device.

[0171] The processing module 601 can be a processor, for example, the processor can be... Figure 2 The processor 201 is located in the middle. The communication module 602 can be a transceiver, transceiver circuit, or communication interface, for example... Figure 2 The communication interface 203 and the storage module 603 can be a memory, for example... Figure 2 The memory 202 in the middle.

[0172] The modules of the aforementioned computing device can also be used to perform other actions in the above method embodiments. All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0173] For more details on how the modules included in the above computing device implement the above functions, please refer to the descriptions in the preceding method embodiments, which will not be repeated here. The various embodiments in this specification are described in a progressive manner; similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments.

[0174] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0175] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0180] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating an AI model, characterized in that, A method applicable to a computing device including a first processor and a second processor, wherein the memory of the first processor is first memory and the memory of the second processor is second memory, the first processor controls the second processor to cause the second processor to execute multiple operators of an AI model to train the AI ​​model, the method comprising: The first processor constructs a target optimization function with operation instruction information as the optimization variable based on the execution time of each of the multiple operators in the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator; wherein, the memory unloading time is the time it takes for the second processor to obtain the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory, and the first operator is any one of the multiple operators; Global optimization is performed on the target optimization function to obtain the operation indication information; the operation indication information is used to indicate whether to perform the target operation for each of the plurality of operators, the target operation including recalculation operation or memory unloading operation; The first processor sends an adjustment instruction to the second processor; the adjustment instruction includes the operation instruction information, and the adjustment instruction is used to instruct the second processor to execute the plurality of operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

2. The method according to claim 1, characterized in that, The objective optimization function takes the execution time of the operator and the memory unloading time as optimization objectives, and takes the memory usage of the second memory during the operation of the operator as a constraint.

3. The method according to claim 2, characterized in that, The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i E represents the memory usage of the second memory during the execution of the i-th operator. total E represents the total memory usage for training the AI ​​model. max Indicates the constraint value of available memory; x i Used to indicate whether to perform the target operation on the i-th operator, wherein when x i When x = 1, the target operation is performed on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

4. The method according to claim 1, characterized in that, The objective optimization function takes the memory usage of the second memory during the operator's operation as the optimization objective, and the execution time of the operator and the memory unloading time as constraints.

5. The method according to claim 4, characterized in that, The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second memory during the execution of the i-th operator. target This indicates the total training time of the AI ​​model; x i Used to indicate whether to perform the target operation on the i-th operator, wherein when x i When x = 1, the target operation is performed on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

6. The method according to claim 1, characterized in that, The objective optimization function takes the execution time of the operator, the memory unloading time, and the memory usage of the second memory during the operation of the operator as optimization objectives, and takes the execution time of the operator, the memory unloading time, and the memory usage as constraints.

7. The method according to claim 6, characterized in that, The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i T represents the execution time of the i-th operator. swap_i E represents the memory unloading time of the i-th operator. i T represents the memory usage of the second memory during the execution of the i-th operator. target E represents the total training time of the AI ​​model. total E represents the total memory usage for training the AI ​​model. max Represents the constraint value of available memory; α and β are weighting coefficients; x i Used to indicate whether to perform the target operation on the i-th operator, wherein when x i When x = 1, the target operation is performed on the i-th operator. i When = 0, the target operation is not performed on the i-th operator.

8. The method according to claim 3, 5, or 7, characterized in that, When x i =1, and min(T) i T swap_i ) = T i When the i-th operator is called, the recalculation operation is performed. When x i =1, and min(T) i ,T swap_i ) = T swap_i At that time, the memory unloading operation is performed on the i-th operator.

9. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first processor determines the execution time of each operator in the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the training process of the AI ​​model.

10. The method according to claim 9, characterized in that, The load information of the plurality of operators includes the memory access information of each of the plurality of operators and / or the cache hit rate of each operator; The first processor determines the execution time of the first operator among the multiple operators of the AI ​​model based on the load information of the multiple operators during the training process of the AI ​​model, including: The first processor determines the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the computational load of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

11. The method according to claim 10, characterized in that, The load information of the plurality of operators also includes at least one of the following: the dependency relationship of the plurality of operators, the execution order of the plurality of operators, or the input-output size of each of the plurality of operators; The first processor determines the memory usage of the second memory during the operation of the first operator among the multiple operators of the AI ​​model, based on the load information of the multiple operators during the training process of the AI ​​model, including: The first processor determines the memory usage of the second memory during the operation of the first operator based on the dependencies between the multiple operators, the execution order of the multiple operators, and the input and output sizes of the first operator.

12. The method according to claim 11, characterized in that, The load information of the plurality of operators also includes the memory usage of each of the plurality of operators; The first processor determines the memory unloading time of the first operator among the multiple operators of the AI ​​model based on the load information of the multiple operators during the training process of the AI ​​model, including: The first processor determines the memory unloading time of the first operator based on the memory usage of the first operator, the bandwidth of the second processor, the bandwidth utilization of the second processor, and the latency of the second processor in executing the first operator.

13. A computing device, characterized in that, It includes a building module, a computing module, and a sending module; The construction module is used to construct a target optimization function with operation instruction information as the optimization variable, based on the execution time of each operator in the multiple operators of the AI ​​model, the memory occupation size of the second memory during the operation of each operator, and the memory unloading time of each operator; wherein, the second memory is the memory of the second processor, the memory unloading time is the time it takes for the second processor to obtain the data required by the second processor to execute the first operator when the data in the second memory is unloaded to the first memory, the first operator is any one of the multiple operators, and the first memory is the memory of the first processor; The calculation module is used to perform global optimization on the target optimization function to obtain the operation indication information; the operation indication information is used to indicate whether to perform the target operation on each of the plurality of operators, the target operation including recalculation operation or memory unloading operation. The sending module is used to send an adjustment instruction to the second processor; the adjustment instruction includes the operation instruction information, and the adjustment instruction is used to instruct the second processor to execute the plurality of operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

14. The computing device according to claim 13, characterized in that, The computing device further includes a determination module; The determining module is used to determine the execution time of each operator in the multiple operators of the AI ​​model, the memory usage of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the training process of the AI ​​model.

15. The computing device according to claim 14, characterized in that, The load information of the plurality of operators includes the memory access information of each of the plurality of operators and / or the cache hit rate of each operator; The determining module is specifically used to determine the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the computational load of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

16. The computing device according to claim 14, characterized in that, The load information of the plurality of operators also includes at least one of the following: the dependency relationship of the plurality of operators, the execution order of the plurality of operators, or the input-output size of each of the plurality of operators; The determining module is specifically used to determine the memory usage of the second memory during the operation of the first operator based on the dependency relationship of the multiple operators, the execution order of the multiple operators, and the input and output size of the first operator.

17. The computing device according to claim 14, characterized in that, The load information of the plurality of operators also includes the memory usage of each of the plurality of operators; The determining module is specifically used to determine the memory unloading time of the first operator based on the memory usage of the first operator, the bandwidth of the second processor, the bandwidth utilization of the second processor, and the latency of the second processor executing the first operator.

18. A computing device, characterized in that, The system includes a memory and at least one processor connected to the memory, the at least one processor including a first processor and a second processor; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the at least one processor, cause the first processor to perform the method as described in any one of claims 1 to 12, and cause the second processor to perform a plurality of operators of an AI model to train the AI ​​model.

19. A computer-readable storage medium, characterized in that, The device stores computer instructions that, when executed on a computer, perform the method as described in any one of claims 1 to 12.

Images