Method and device for generating AI model

Abstract

The invention provides a method and a device for generating an AI model, relates to the technical field of artificial intelligence, and aims to improve the efficiency of generating the AI model to a certain extent and reduce resource consumption. The method comprises the steps that a first processor constructs a target optimization function with operation indication information as an optimization variable based on the execution duration of each operator in a plurality of operators of an AI model and the memory occupation size and memory unloading duration of a second memory of a second processor in the operation process of each operator; global optimization is carried out on the target optimization function, operation indication information used for indicating whether target operation is executed on each operator in the multiple operators or not is obtained, and the target operation comprises re-calculation operation or memory unloading operation; and sending an adjustment instruction including the operation instruction information to the second processor to instruct the second processor to execute the plurality of operators according to the operation instruction information so as to train the AI model to obtain a target AI model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a method and device for generating an AI model. Background Art

[0002] With the continuous improvement of AI chip computing power, the increase of communication bandwidth and the continuous optimization of AI model training methods, training high-performance AI models on large-scale device clusters containing AI chips has become an important research hotspot in academia and industry.

[0003] Due to the complex structure of AI models (for example, a large number of network layers) and a large number of parameters, some existing AI model training methods generally have the problem of taking a long time and consuming a large amount of resources (such as computing power and storage resources). Summary of the invention

[0004] The embodiments of the present application provide a method and device for generating an AI model, which can improve the efficiency of generating the AI ​​model and reduce resource consumption to a certain extent.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] In the first aspect, an embodiment of the present application provides a method for generating an AI model, which is applied to a computing device including a first processor and a second processor, wherein the memory of the first processor is the first memory, and the memory of the second processor is the second memory, and the first processor is used to control the second processor to execute multiple operators of the AI ​​model to train the AI ​​model, and the method includes: the first processor constructs a target optimization function with operation indication information as an optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy size of the second memory during the operation of each operator, and the memory unloading time of each operator; and performs global optimization on the target optimization function to obtain operation indication information; the operation indication information is used to indicate whether to perform a target operation on each of the multiple operators, and the target operation includes a recalculation operation or a memory unloading operation; and sends an adjustment instruction to the second processor, the adjustment instruction includes the operation indication information, and the adjustment instruction is used to instruct the second processor to execute multiple operators according to the operation indication information to train the AI ​​model to obtain the target AI model. Among them, the memory unloading time is the time when the second processor obtains the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory, and the first operator is any one of the multiple operators

[0007] In the AI ​​model generation method provided in the embodiment of the present application, the first processor can model the optimization strategy (recalculation or memory unloading) according to the execution time of the operator, the memory occupancy size of the second memory (the memory of the second processor) during the operation of the operator, and the memory unloading time of the operator. In this way, it is possible to choose between the recalculation operation and the memory unloading operation, that is, in the AI ​​model training process, flexibly and dynamically select the appropriate operation according to the actual situation of the operator of the AI ​​model, so as to overcome the problems of large resource consumption and low training efficiency caused by the use of recalculation or memory unloading technology alone, and can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0008] In one possible implementation, the AI ​​model generation method provided in an embodiment of the present application also includes: the first processor determines the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the AI ​​model training process.

[0009] Among them, the operator load information includes at least one of the following: the dependency relationship between multiple operators of the AI ​​model, the execution order of multiple operators, the memory access information of each of the multiple operators, the cache hit rate of each operator or the input and output size of each operator, and the memory usage of each operator.

[0010] In one possible implementation, the first processor determines the execution time of a first operator among multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, including: the first processor determines the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the calculation amount of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0011] In one possible implementation, the first processor determines the memory occupancy of the second memory during the operation of the first operator among multiple operators of the AI ​​model according to the load information of multiple operators during the training of the AI ​​model, including: the first processor determines the memory occupancy of the second memory during the operation of the first operator according to the dependency relationship of the multiple operators, the execution order of the multiple operators, and the input and output size of the first operator.

[0012] In one possible implementation, the load information of multiple operators of the AI ​​model also includes the memory occupancy size of each of the multiple operators. The above-mentioned first processor determines the memory unloading duration of the first operator among the multiple operators of the AI ​​model according to the load information of the multiple operators during the AI ​​model training process, including: the first processor determines the memory unloading duration of the first operator according to the memory occupancy size of the first operator, the bandwidth of the second processor, the bandwidth utilization rate of the second processor, and the latency of the second processor to execute the first operator.

[0013] The above-mentioned operator execution time, the memory usage of the second memory during the operation of the operator, and the memory unloading time are indicators related to the hardware capabilities of the device and the characteristics of the operator. These indicators are taken into consideration when modeling the optimization strategies for operator recalculation and memory unloading, so as to avoid the problem of low efficiency and high resource consumption in the training process of the AI ​​model due to selecting a certain optimization strategy based only on the theoretical operator calculation amount and memory usage.

[0014] In a second aspect, an embodiment of the present application provides a computing device, including a construction module, a calculation module, and a sending module. The construction module is used to construct a target optimization function with operation instruction information as an optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy size of the second memory during the operation of each operator, and the memory unloading time; wherein the second memory is the memory of the second processor, the memory unloading time is the time for the second processor to obtain the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory, the first operator is any one of the multiple operators, and the first memory is the memory of the first processor. The calculation module is used to perform global optimization on the target optimization function to obtain operation instruction information, which is used to indicate whether to perform a target operation on each of the multiple operators, and the target operation includes a recalculation operation or a memory unloading operation. The sending module is used to send an adjustment instruction to the second processor, the adjustment instruction includes operation instruction information, and the adjustment instruction is used to instruct the second processor to execute multiple operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

[0015] In one possible implementation, the computing device also includes a determination module, which is used to determine the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the training process of the AI ​​model.

[0016] Among them, the operator load information includes at least one of the following: the dependency relationship between multiple operators of the AI ​​model, the execution order of multiple operators, the memory access information of each of the multiple operators, the cache hit rate of each operator or the input and output size of each operator, and the memory usage of each operator.

[0017] In a possible implementation, the above-mentioned determination module is specifically used to determine the execution time of the first operator based on the memory access information of the first operator, the cache hit rate of the first operator, the calculation amount of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0018] In a possible implementation, the determination module is specifically used to determine the memory usage size of the second memory during the operation of the first operator according to the dependency relationship of multiple operators, the execution order of multiple operators, and the input and output size of the first operator.

[0019] In one possible implementation, the load information of multiple operators of the AI ​​model also includes the memory usage of each of the multiple operators; the above-mentioned determination module is specifically used to determine the memory unloading duration of the first operator based on the memory usage of the first operator, the bandwidth utilization bandwidth of the second processor, the second processor, and the latency of the second processor to execute the first operator.

[0020] In the above-mentioned first aspect and second aspect, the target optimization function takes the execution time and memory unloading time of the operator as optimization targets, and takes the memory occupancy size of the second memory during the operation of the operator as a constraint condition.

[0021] In a possible implementation, the above AI model includes n operators, where n is an integer greater than or equal to 2; the target optimization function is:

[0022]

[0023]

[0024] Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second memory during the operation of the i-th operator, E total Indicates the total memory usage of the AI ​​model training, E max Indicates the constraint value of available memory; x i Used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i=0, the target operation is not performed on the i-th operator.

[0025] The optimization goal of the above objective optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the additional time overhead caused by recalculating the operator or unloading the memory is minimized, that is, to ensure optimal performance. The constraints of the target optimization function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the actual memory size occupied by training the AI ​​model does not exceed the size of the available memory constrained by the AI ​​chip.

[0026] In the above-mentioned first aspect and second aspect, the target optimization function takes the memory usage size of the second memory during the operation of the operator as the optimization target, and takes the execution time of the operator and the memory unloading time as the constraints.

[0027] In a possible implementation, the above AI model includes n operators, where n is an integer greater than or equal to 2; the target optimization function is:

[0028]

[0029] Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second memory during the operation of the i-th operator, T target Indicates the total time of training the AI ​​model; x i Used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

[0030] The optimization goal of the above objective optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the memory saved by recalculating the operator or memory unloading operations is maximized, that is, the memory consumed by the AI ​​model training process is minimized.

[0031] The constraints of the above objective optimization function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the total time consumed for training the AI ​​model does not exceed the estimated total time, where the total time consumed for training the AI ​​model is the sum of the total execution time of all operators and the additional time consumed due to performing heavy calculation operations or memory unloading operations.

[0032] In the above-mentioned first aspect and second aspect, the target optimization function takes the execution time of the operator, the memory unloading time and the memory occupancy size of the second memory during the operation of the operator as the optimization target, and takes the execution time of the operator, the memory unloading time and the memory occupancy size of the second memory during the operation of the operator as the constraint conditions.

[0033] In a possible implementation, the AI ​​model includes n operators, where n is an integer greater than or equal to 2; the target optimization function is:

[0034]

[0035] Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second memory during the operation of the i-th operator, T target Indicates the total time of training the AI ​​model, E total Indicates the total memory usage of the AI ​​model training, E max represents the constraint value of available memory; α, β are weight coefficients; x i Used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

[0036] The optimization goal of the above objective optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the memory saved by recalculating the operator or unloading the memory is maximized, and the additional time overhead caused by recalculating the operator or unloading the memory is minimized, that is, to ensure that the AI ​​model training process consumes the least memory and has the best performance.

[0037] The constraints of the above objective optimization function are: and The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent on training the AI ​​model does not exceed the estimated total time, and the actual memory size occupied by training the AI ​​model does not exceed the size of the available memory constrained by the AI ​​chip.

[0038] In one possible implementation, when x i =1, and min(T i ,T swap_i )=T i When x i =1, and min(Ti ,T swap_i )=T swap_i , the memory unloading operation is performed on the i-th operator.

[0039] In a third aspect, an embodiment of the present application provides a computing device comprising a memory and at least one processor connected to the memory, the at least one processor comprising a first processor and a second processor; the memory is used to store computer program code, the computer program code comprising computer instructions, and when the computer instructions are executed by at least one processor, the first processor executes the method described in the first aspect and any one of its possible implementation methods, and the second processor executes multiple operators of an AI model to train the AI ​​model.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer instructions, which, when executed on a computer, execute the method of the first aspect and any one of its possible implementations.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are run on a computer, the method of the first aspect and any one of its possible implementations is executed.

[0042] In a sixth aspect, an embodiment of the present application provides a chip system, comprising: a processor, configured to call and run a computer program from a memory, so that a terminal device equipped with the chip system executes the method of the first aspect and any one of its possible implementation methods.

[0043] It should be understood that the beneficial effects achieved by the technical solutions of the second to sixth aspects of the present application and the corresponding possible implementation methods can be referred to the technical effects of the first aspect and its corresponding possible implementation methods mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the structure of a Transformer model provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application;

[0046] Figure 3 One of the schematic diagrams of a method for generating an AI model provided in an embodiment of the present application;

[0047] Figure 4 A second schematic diagram of a method for generating an AI model provided in an embodiment of the present application;

[0048] Figure 5One of the structural schematic diagrams of a computing device provided in an embodiment of the present application;

[0049] Figure 6 The second structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0051] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first processor and a second processor are used to distinguish different processors rather than to describe a specific order of the processors.

[0052] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0053] In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two, and "plurality" may also be described as at least two.

[0054] First, some concepts involved in the embodiments of the present application are explained.

[0055] 1. AI Model

[0056] In the field of artificial intelligence technology, a mathematical model is constructed to achieve a certain function (such as target detection, image processing, natural language processing, etc.). The process of using an AI model to achieve a certain function includes model training and model reasoning of the AI ​​model. Model training can obtain an AI model with better performance, and model reasoning is to use the trained AI model to achieve specific functions.

[0057] The AI ​​model includes the model architecture and model parameters. Usually, after the AI ​​model architecture is designed, the initial AI model can be continuously trained based on a large number of training samples using machine learning (such as deep learning) methods (the training process can be understood as the process of continuously updating the parameters of the AI ​​model) to obtain an AI model that can realize specific functions. The training process of the AI ​​model includes: inputting the training samples into the current AI model, and obtaining the prediction results through the processing of the AI ​​model (i.e. mathematical operations); then comparing the prediction results with the actual results of the training samples, and determining the model loss, and then updating the parameters of the AI ​​model according to the model loss. By cyclically executing the above process, an AI model with better performance is obtained.

[0058] Optionally, commonly used AI models may include but are not limited to the following models: convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM) network, Transformer network.

[0059] Among them, CNN is a deep learning model for image recognition and classification. RNN is a recursive neural network used to process sequence data (such as text, speech). It maintains historical information through the loop layer to understand the long-term dependencies of the data. LSTM is a variant of the recursive neural network (RNN). The special thing about LSTM is that it has a memory unit that can store long-term information inside, which enables it to handle long-term dependency problems, has the ability of long-term memory, and can remember the long-term dependencies of data. Transformer is a neural network model based on the attention mechanism. It captures the relationship between data through the attention mechanism and can be used for processing sequence data (such as speech recognition, machine translation, and text classification, etc.).

[0060] It is understandable that AI models actually perform a large number of complex mathematical operations, that is, AI models include a large number of operators (operators can be understood as mappings, mapping operations from one function space to another function space). The operators in the AI ​​model can realize the functions of the AI ​​model by executing them according to certain dependencies and execution orders. Among the multiple operators in the AI ​​model, some operators have a small amount of calculation, while some operators have a large amount of calculation.

[0061] For example, taking the above Transformer as an example, Figure 1 A schematic diagram of the structure of a Transformer is shown. Figure 1As shown in Figure 1, Transformer includes multiple Transformer blocks, which are connected in series, that is, the output of the previous Transformer block serves as the input of the next Transformer block. The structure of each Transformer block can be referred to Figure 1 , where the linear module (Linear) is used to linearly transform the Self-attention input (i.e., the input information of the Transformer) to obtain three matrices, Q (Query), K (Key), and V (Value); then, Q and K are dot-multiplied (inner product); Softmax is used to normalize the dot-multiplication results of Q and K; Dropout is used to randomly discard the normalized results to improve the robustness of the Transformer; finally, after linear processing of the Self-attention output (self-attention output module) and processing of the feed-forward network (FFN), a Transformer block output is obtained.

[0062] Combination Figure 1 , the Linear, matrix dot product, and FFN operations in the above Transformer block are matrix operation operators, and the remaining Softmax, Dropout, etc. are all vector operation operators. It can be understood that, in general, the amount of calculation of vector operation operators is small, the amount of calculation of matrix operation operators is large, and the time consumption of vector operation operators is less than that of matrix operation operators. However, in some cases, due to hardware performance reasons, the amount of calculation of vector operation operators may be greater than that of matrix operation operators, thus causing the time consumption of vector operation operators to be greater than that of matrix operation operators. For example, as the length of the vector (sequence length) increases, the computational complexity of Softmax (vector operation operator) in Transformer increases quadratically, and the time overhead of Softmax operation of very long sequences may also be very large, which may be higher than the time overhead of FFN (matrix operation operator) operation.

[0063] 2. AI chips

[0064] AI chips are modules specifically used to process large numbers of computing tasks in the field of artificial intelligence. AI chips can be called AI accelerators, AI wafers, computing cards or heterogeneous acceleration units, etc. AI accelerators are also a type of processor.

[0065] Exemplarily, AI chips may include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and the like.

[0066] GPU, also known as visual processor or display chip, is a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.). GPU contains thousands of cores, can process thousands of threads at the same time, and can run a large number of calculations in parallel.

[0067] FPGA is a programmable logic processor that users can repeatedly program according to their needs. FPGA is good at processing short and intensive data access. FPGA allows users to program circuit paths through tiny logic blocks and can process any type of digital function.

[0068] ASIC is an integrated circuit with special specifications designed for a specific purpose. It is designed and manufactured according to the requirements of specific users and specific electronic systems. For example, tensor processing unit (TPU) and neural network processing unit (NPU) are both ASICs. TPU and NPU are chips designed for machine learning, suitable for the calculation of massive data, and have powerful computing capabilities.

[0069] Based on the above conceptual description, at present, when training AI models based on AI chips (such as GPUs), due to the complex structure of the AI ​​models (such as a large number of network layers) and the large number of parameters, training AI models poses certain challenges to the computing power and storage of AI chips.

[0070] At present, one method to optimize the training process of the AI ​​model is to recompute all or part of the operators of the AI ​​model, where recomputing all operators can be called full recomputed, and recomputing part of the operators can be called selective recomputed. It can be understood that recomputation is a technology that divides the calculation process into multiple small blocks, each small block only uses a part of the data for calculation, caches part of the data (such as the output result of the AI ​​model), and discards part of the data (such as some intermediate calculation results). In this way, in the subsequent calculation process, only the part of the data that has not been cached needs to be recalculated, and then the cached data is merged with the recalculated data. On the one hand, the recomputation technology can reduce the memory usage of the AI ​​chip (such as the GPU's video memory), thereby allowing larger AI models to be trained based on smaller video memory. On the other hand, in some cases, the recomputation technology can also increase the computing throughput.

[0071] Full recalculation means that only some inputs and outputs of the model are cached, and the data in the intermediate operation process of the model (or called activation variables) are not cached. In the subsequent reverse calculation process (updating model parameters through reverse calculation), all uncached data needs to be recalculated. Full recalculation will bring additional computing power overhead (consuming about 30%-40% additional computing power). For example, for the above Figure 1 For the large language model of the Transformer architecture shown in the figure, full retiming is used, the data output by each Transformer block is cached, and the activation variables in the intermediate calculation process of the Transformer block are discarded.

[0072] Selective recomputation refers to caching some inputs and outputs of the model, caching some data of the intermediate calculation process, and discarding some data of the intermediate calculation process, that is, selective recomputation only repeats the calculation of part of the data. Specifically, based on the theoretical amount of calculation, it is possible to selectively repeat the calculation of a part of the data with low computational complexity but large memory usage of the AI ​​chip. In this way, the memory usage of the AI ​​chip during AI model training can be reduced, and the training speed and effect of the AI ​​model can be improved while ensuring that the memory usage of the AI ​​chip is not too high. For example, for the above Figure 1 For the large language model of the Transformer architecture shown in the figure, in general, the Softmax and Dropout operations occupy a large amount of memory, but the computational complexity of Softmax and Dropout is low. Therefore, Softmax and Dropout can be set as recalculation objects, while FFN does not need to be recalculated.

[0073] Another method to optimize the training process of AI models is parameter offload technology. Parameter swap is actually memory unloading (swap), that is, the intermediate variables and training parameters of the model are stored in the memory of the CPU instead of the memory of the AI ​​chip to reduce the memory usage of the AI ​​chip. During the training process of the AI ​​model, the intermediate variables and training parameters of the AI ​​model are divided into multiple blocks, and each block is assigned to a different AI chip for calculation. When an AI chip needs to access the intermediate variables and training parameters on other AI chips, the AI ​​chip can obtain them from the memory of the CPU through the network instead of from the memory of other AI chips. Memory offloading can reduce the memory usage of the AI ​​chip, so that larger models can be trained based on the limited memory of the AI ​​chip. However, the bandwidth from the CPU to the AI ​​chip (i.e., host to device, H2D) is small, and the speed at which the AI ​​chip obtains intermediate variables and training parameters from the CPU is slow and time-consuming. Therefore, the above-mentioned parameter swap technology will cause a great degradation in performance while obtaining the memory benefits of the AI ​​chip. It can be understood that the above-mentioned performance mainly refers to the time overhead or speed of training the AI ​​model, and the time overhead or speed factor affects the training efficiency of the AI ​​model.

[0074] It should be noted that since the essence of parameter offloading is memory offloading, in order to facilitate understanding and description, parameter exchange is described as memory offloading in the following embodiments.

[0075] In summary, some existing methods for training AI models have the problem of being time-consuming and consuming a large amount of resources (such as computing power resources and storage resources). In response to the above problems, an embodiment of the present application provides a method for generating an AI model, which is applied to a computing device including a first processor and a second processor, wherein the second processor is used to execute multiple operators of the AI ​​model to train the AI ​​model, and the first processor is used to perform the following process: the first processor constructs a target optimization function with operation indication information as the optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy size of the second memory during the operation of each operator, and the memory unloading time of each operator; and performs global optimization on the target optimization function to obtain operation indication information, which is used to indicate whether to perform a target operation on each of the multiple operators, and the target operation includes a recalculation operation or a memory unloading operation; further, the first processor sends an adjustment instruction to the second processor, the adjustment instruction includes the operation indication information, and the adjustment instruction is used to instruct the second processor to execute multiple operators according to the operation indication information to train the AI ​​model to obtain the target AI model. Among them, for any one operator among the multiple operators (referred to as the first operator), the memory unloading duration is the duration for the second processor to obtain the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory. The first memory is the memory of the first processor, and the second memory is the memory of the second processor.

[0076] The AI ​​model generation method provided in the embodiment of the present application can choose between recalculation operations and memory offloading operations, that is, in the AI ​​model training process, flexibly and dynamically select appropriate operations according to the actual situation of the operator of the AI ​​model, so as to overcome the problems of large resource consumption and low training efficiency caused by the use of recalculation or memory offloading technology alone, and can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0077] Optionally, the first processor in the above computing device may be a general-purpose central processing unit (CPU) or other general-purpose processor, and the second processor may be an AI chip, such as a GPU, NPU, TPU, etc. The CPU is mainly responsible for the logical calculation and logical control functions of the load, and the AI ​​chip is mainly responsible for a large number of complex computing functions.

[0078] For example, Figure 2 A schematic diagram of the hardware structure of a computing device provided in an embodiment of the present application is provided. Figure 2 The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits. Figure 2As shown, the computing device may include: a processor 201, a memory 202, a communication interface 203, and an AI accelerator 205. The processor 201, the memory 202, the communication interface 203, and the AI ​​accelerator 205 may be connected to each other via a bus 204, or in other ways.

[0079] The processor 201 is the control center of the computing device. The processor 201 may be a CPU or other general-purpose processors. The general-purpose processor may be a microprocessor or any conventional processor.

[0080] The controller in the processor 201 is the nerve center and command center of the computing device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions. Optionally, a memory can also be set in the processor 201 to store instructions and data. Exemplarily, the processor 201 may include one or more CPUs, such as Figure 2 CPU 0 and CPU 1 are shown in .

[0081] The memory 202 includes, but is not limited to, a random access memory (RAM), a read only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or an optical memory, a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. In the embodiment of the present application, the memory 202 can store information such as computer instructions.

[0082] In a possible implementation, the memory 202 may exist independently of the processor 201. The memory 202 may be connected to the processor 201 via a bus 204 and used to store data, instructions, or program codes. When the processor 201 calls and executes the instructions or program codes stored in the memory 202, the relevant steps in the method provided in the embodiment of the present application can be implemented.

[0083] In another possible implementation, the memory 202 may also be integrated with the processor 201 .

[0084] The communication interface 203 may be a transceiver module, which is used to communicate with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. The communication interface 203 may receive instructions, messages or data, etc. The transceiver module may be a device such as a transceiver or a transceiver. Optionally, the communication interface 203 may also be a transceiver circuit located in the processor 201, which is used to implement signal input and signal output of the processor. The communication interface 203 may be a wired interface (port), such as a fiber distributed data interface (FDDI), a gigabit Ethernet (GE) interface, or the communication interface 203 may also be a wireless interface.

[0085] The AI ​​accelerator 205 is a processor that is heterogeneous with the processor 201. The AI ​​accelerator 205 is a module dedicated to processing a large number of computing tasks in the field of artificial intelligence. The AI ​​accelerator 205 is also connected to other components through the bus 204. The AI ​​accelerator 205 may include a GPU, an FPGA, an ASIC, etc. For more detailed description of the AI ​​accelerator 205, please refer to the content of the concept introduction part of the above embodiment, which will not be repeated here.

[0086] The processor 202 (i.e., central processing unit CPU) is a general computing module, which is mainly responsible for the logical calculation and logical control functions of the load, and can efficiently process single complex calculation sequence tasks, but its performance in large-scale calculations is relatively low. Therefore, the processor can distribute large-scale calculation tasks to the AI ​​accelerator 205, and after the AI ​​accelerator 205 completes the calculation, it returns the calculation results to the processor 201.

[0087] The bus 204 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus may also be divided into a serial bus and a parallel bus. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0088] It should be noted that Figure 2 The computing device shown is only one example of a computing device that can have more than Figure 2More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components.

[0089] Based on the above, an embodiment of the present application provides a method for generating an AI model, which can be executed by a first processor (such as a CPU) in a computing device. The first processor executes the scheme to generate an optimization strategy for adjusting the AI ​​model training process, and controls the second processor to execute multiple operators of the AI ​​model with the optimization strategy to train the AI ​​model, thereby generating a target AI model; wherein the memory of the first processor is called the first memory, and the memory of the second processor is called the second memory. Figure 3 As shown, the method for generating an AI model provided in an embodiment of the present application includes S301-S303.

[0090] S301. The first processor constructs a target optimization function with operation instruction information as the optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time.

[0091] It should be understood that the AI ​​model includes multiple operators, and running the AI ​​model is the process of executing the multiple operators according to the operator call relationship and execution order of the multiple operators of the AI ​​model. The execution time of each of the multiple operators, the memory occupancy size of the second memory during the operation of each operator, and the memory unloading time are obtained based on the iteration result of any one of the training processes (i.e., multiple iterations) in which the second processor executes the training process of the AI ​​model multiple times.

[0092] For any operator in the AI ​​model (hereinafter referred to as the first operator), the execution time of the first operator is the time required for the second processor to execute the first operator (that is, the time required for the forward calculation of the first operator). The execution time of the first operator can also be referred to as the operator performance of the first operator. The memory occupancy size of the second memory during the operation of the first operator is the size of the storage space currently occupied by the second memory during the execution of the first operator by the second processor. The memory unloading time is the time required for the second processor to obtain the data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory. The memory unloading time of the first operator can also be referred to as the memory unloading cost of the first operator.

[0093] Optionally, the execution order of some operators among the multiple operators in the AI ​​model may be serial execution, and the execution order of some operators may be parallel execution. In combination with the example, it should be noted that for the case of serial execution, the first operator is currently executed, and the memory occupancy of the second memory during the operation of the first operator is the memory required to run the first operator. For the case of parallel execution, when the first operator is executed, the second operator executed in parallel with the first operator is also executed, then the memory occupancy of the second memory during the operation of the first operator is the sum of the memory required to run the first operator and the memory required to run the second operator.

[0094] It can be understood that if the first operator is recalculated, then during the reverse calculation process, the time required to recalculate the first operator is the execution time of the first operator, that is, the additional time consumed to recalculate the first operator.

[0095] The memory usage size of the second memory during the operation of the first operator is the size of the storage space occupied in the second memory during the normal execution of the first operator by the second processor (the second processor and the memory of the second processor participate in the operator calculation, and the memory unloading technology is not applied). It can be understood that if the first operator is recalculated, the relevant data of the first operator can be discarded during the forward calculation process, so that the memory occupied by the first operator can be saved, that is, the memory usage size of the second memory is the memory size that can be saved when the first operator is recalculated. The memory unloading duration is the additional time consumed in the process of executing the first operator when the memory unloading technology is used.

[0096] Based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator, it is comprehensively determined whether the execution process of each of the multiple operators needs to be optimized, that is, whether to perform a recalculation operation or a memory unloading operation on the first operator; and it is determined what kind of operation optimization is used for the first operator, that is, whether to use a recalculation operation for optimization or a memory unloading operation for optimization. In an embodiment of the present application, to achieve this purpose, an optimization function is constructed based on the execution time, the memory occupancy size, and the memory unloading time. The optimization variable of the optimization function is the operation indication information for optimizing the execution process of each operator. The operation indication information is used to indicate whether to perform a target operation on each of the multiple operators of the AI ​​model, and the target operation includes a recalculation operation or a memory unloading operation.

[0097] Optionally, the target optimization function constructed based on the execution time of the operator, the memory usage of the second memory during the operation of the operator, and the memory unloading time may include the following three functions.

[0098] The first type: The target optimization function takes the execution time and memory unloading time of the operator as the optimization target, and takes the memory usage of the second memory during the operation of the operator as the constraint condition.

[0099] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the target optimization function may be a function shown in the following formula (1):

[0100]

[0101] In formula (1), T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second memory during the operation of the i-th operator, E total Indicates the total memory usage of the AI ​​model training, E max Indicates the constraint value of available memory. E total is the theoretical estimated total memory usage, E max It is the minimum value of available memory during the training of AI models. It should be understood that E max It is related to the hardware parameters of the AI ​​chip. For different AI chips, E max May be different.

[0102] x in formula (1) i Used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

[0103] Combined with the above formula (1), the optimization goal of the target optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the additional time overhead caused by recalculating the operator or unloading the memory is minimized, that is, to ensure optimal performance. The constraints of the target optimization function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the actual memory size occupied by training the AI ​​model does not exceed the size of the available memory constrained by the AI ​​chip.

[0104] The second type: The target optimization function takes the memory usage of the second memory during the operation of the operator as the optimization target, and takes the execution time of the operator and the memory unloading time as the constraints.

[0105] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the target optimization function may be a function shown in the following formula (2):

[0106]

[0107] In formula (2), T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second inner ability during the operation of the i-th operator, T target Indicates the total time required to train the AI ​​model (theoretically estimated total time).

[0108] Similarly, x in formula (2) i Used to indicate whether to perform the target operation on the i-th operator. i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

[0109] Combined with the above formula (2), the optimization goal of the target optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the memory saved by recalculating the operator or unloading the memory is maximized, that is, the memory consumed by the AI ​​model training process is minimized. The constraints of the target optimization function are: The meaning of this optimization condition is: during the training process of the AI ​​model, the total time consumed for training the AI ​​model does not exceed the estimated total time, where the total time consumed for training the AI ​​model is the sum of the total execution time of all operators and the additional time consumed due to performing heavy calculation operations or memory unloading operations.

[0110] The third type: The target optimization function takes the execution time, memory unloading time and memory usage of the operator as the optimization target, and takes the execution time, memory unloading time and memory usage of the operator as the constraint conditions.

[0111] The AI ​​model includes n operators, where n is an integer greater than or equal to 2. In some embodiments, the target optimization function may be the function shown in formula (3):

[0112]

[0113] In formula (3), T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i Indicates the memory usage of the second memory during the operation of the i-th operator, T target Indicates the total time of training the AI ​​model, E total Indicates the total memory usage of the AI ​​model training, Emax Represents the constraint value of available memory; α and β are weight coefficients, which are used to unify the dimensions of duration and memory and balance the importance of duration and memory in the optimization objective.

[0114] Similarly, x in formula (3) i Used to indicate whether to perform the target operation on the i-th operator. i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

[0115] Combined with the above formula (3), the optimization goal of the target optimization function is The meaning of this optimization goal is: during the training process of the AI ​​model, the memory saved by recalculating the operator or unloading the memory is maximized, and the additional time overhead caused by recalculating the operator or unloading the memory is minimized, that is, the memory consumed by the AI ​​model training process is minimized and the performance is optimal. The constraints of the objective optimization function are: and The meaning of this optimization condition is: during the training process of the AI ​​model, the total time spent on training the AI ​​model does not exceed the estimated total time, and the actual memory size occupied by training the AI ​​model does not exceed the size of the available memory constrained by the AI ​​chip.

[0116] S302: Perform global optimization on the target optimization function to obtain operation indication information for indicating whether to perform a target operation on each of the multiple operators, where the target operation includes a recalculation operation or a memory unloading operation.

[0117] It should be noted that when it is necessary to perform a target operation on the i-th operator, the target operation performed on the i-th operator is the operation that consumes the shortest time between the recalculation operation and the memory unloading operation. That is, when x i =1, and min(T i ,T swap_i )=T i When x i =1, and min(T i ,T swap_i )=T swap_i , performing the memory unloading operation on the i-th operator.

[0118] An optimization algorithm can be used to solve the above-mentioned target optimization function. In the embodiment of the present application, the performance gains and losses brought about by memory unloading and recalculation are considered in a coordinated manner, and an optimization algorithm is used to perform global optimization on the target function to obtain the global optimal solution.

[0119] For the first objective optimization function shown in the above formula (1), the objective optimization function can be transformed into a 0-1 knapsack problem, and a dynamic programming algorithm can be used to solve the objective optimization function.

[0120] Specifically, the transformation process of formula (1) is as follows:

[0121]

[0122] It should be understood that the prototype of the 0-1 knapsack problem is: there are n items and a knapsack with a maximum capacity (weight of the items) of w, and the weight of any item is w i , the value is v i , find out which items should be loaded into the backpack to maximize the total value of the items in the backpack. For an item, the loading state of the item includes loading into the backpack or not loading into the backpack, that is, the loading state of an item can be 0 and 1, 0 means not loaded, 1 means loaded, then the loading state of item i can be set to x i , x i ∈(0,1).

[0123] The objective optimization function of the above knapsack problem is:

[0124]

[0125] The 0-1 knapsack problem can be solved by a dynamic programming method. The principle of the dynamic programming method is to transform the multi-stage decision-making process into a series of single-stage decision-making problems, and use the recursive relationship between the stages to determine the optimal decision for each stage one by one, and finally stack up the optimal decision results of the multi-stage decisions. For the detailed solution process of using the dynamic programming method to solve the 0-1 knapsack problem, please refer to the existing technical information, and the embodiments of this application will not be described in detail.

[0126] Referring to the description of the knapsack problem, in the embodiment of the present application, whether each operator executes the target operation is equivalent to whether each item in the knapsack problem is put into the knapsack. Therefore, the execution time, memory usage and memory unloading time of the operator are modeled as a 0-1 knapsack problem, and x is obtained by dynamic programming. i , so that it can be determined whether each operator needs to perform the target operation and which operation to perform (recomputation operation or memory unloading operation).

[0127] For the second objective optimization function shown in the above formula (2), the form of the objective optimization function is a standard 0-1 knapsack problem, so the dynamic programming algorithm can be directly used to solve the objective optimization function.

[0128] For the third objective optimization function shown in formula (3), the coefficients α and β are used in formula (3) to unify the dimensions of memory usage and duration, and the objective problem is transformed into a single objective problem, which can then be transformed into a knapsack problem for solution. The specific algorithm reasoning process can refer to existing technical materials and will not be described in detail here.

[0129] S303: The first processor sends an adjustment instruction to the second processor, where the adjustment instruction includes operation instruction information.

[0130] The adjustment instruction is used to instruct the second processor to execute multiple operators according to the operation instruction information to train the AI ​​model to obtain the target AI model.

[0131] In the embodiment of the present application, the operation instruction information obtained by the first processor in solving the target optimization function is an optimization strategy for optimizing the training process of the AI ​​model. The first processor can send an adjustment instruction containing the operation instruction information to the second processor, so that the second processor performs corresponding operations on one or more operators according to the operation instruction information. For example, if the operation instruction information indicates that the target operation of the i-th operator is a recalculation operation, then when the second processor executes the i-th operator, it performs a recalculation operation on the i-th operator.

[0132] In one implementation, the first processor can monitor the current task running status of the AI ​​model (including the running status of each operator) and issue adjustment instructions according to the running status of the operator. Specifically, the computing device may include a monitoring module and an instruction issuing module for monitoring the running status of the current task. The monitoring module is responsible for monitoring the current task running status and feeding back the execution status of the operator to the instruction issuing module (for example, through TaskScheduler). The instruction issuing module sends adjustment instructions to the second processor at a specific time point based on the running status of the operator fed back by the monitoring module. Optionally, the monitoring module and the instruction issuing module can communicate with the instruction issuing module through inter-process communication (IPC). The monitoring module and the instruction issuing module can also be hardened hardware modules that communicate through hardware interfaces.

[0133] In summary, in the AI ​​model generation method provided in the embodiment of the present application, the optimization strategy (recalculation or memory unloading) can be modeled according to the execution time of the operator, the memory occupancy size of the second memory during the operation of the operator, and the memory unloading time. In this way, it is possible to choose between the recalculation operation and the memory unloading operation, that is, in the AI ​​model training process, according to the actual situation of the operator of the AI ​​model, the appropriate operation can be flexibly and dynamically selected to overcome the problems of large resource consumption and low training efficiency caused by the use of recalculation or memory unloading technology alone, which can improve the efficiency of generating AI models and reduce resource consumption to a certain extent.

[0134] Optionally, combined Figure 3 ,like Figure 4 As shown, before the above-mentioned S301, the method for generating an AI model provided in the embodiment of the present application also includes S300.

[0135] S300. The first processor determines the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the execution of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the AI ​​model training process.

[0136] Optionally, the load information of multiple operators during the AI ​​model training process (also referred to as operator load characteristics) is obtained from the iteration results of any one of the training processes in which the second processor executes the AI ​​model training process multiple times (i.e., multiple iterations).

[0137] The operator load information includes at least one of the following: the dependency relationship between multiple operators of the AI ​​model, the execution order of multiple operators, the memory access information of each of the multiple operators, the cache hit rate of each operator, the input and output size of each operator, or the memory usage of each operator.

[0138] The dependency relationship among multiple operators refers to the dependency relationship between the inputs and outputs of multiple operators. For example, if the input of one operator is the input of another operator, then the two operators have a dependency relationship.

[0139] The execution order of operators can include parallel or serial order.

[0140] The memory access information of an operator refers to the amount of data accessed by cache units at different levels.

[0141] The cache hit rate of an operator is the ratio of the number of hits to the total number of accesses (the sum of the number of hits and the number of misses) when reading data from the cache during the execution of the operator. If the required data can be accessed, it is a hit; if the required data cannot be accessed, it is a miss.

[0142] The input and output size of an operator includes the input size and output size of the operator.

[0143] The memory size of an operator is the size of the memory of the second processor occupied by executing the operator.

[0144] In one implementation, the first processor determines the execution time of the first operator among multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, which specifically includes S300a.

[0145] S300a, the first processor determines the execution time of the first operator according to the memory access information of the first operator, the cache hit rate of the first operator, the calculation amount of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

[0146] In one implementation, the process of determining the execution time of the first operator may include the following steps:

[0147] Step 1: Determine the data handling volume, bandwidth utilization, and instruction execution efficiency during the execution of the first operator according to the memory access information and cache hit rate of the first operator.

[0148] The data handling volume of the first operator is the ratio of the memory access information of the first operator to the cache hit rate of the first operator.

[0149] Step 2: Determine the calculation time of the first operator according to the calculation amount of the first operator, the computing power of the second processor, and the instruction execution efficiency of the first operator; determine the data transfer time of the first operator according to the data transfer amount of the first operator, the bandwidth utilization rate, and the bandwidth of the second processor.

[0150] For ease of understanding, the computational effort of the first operator is F n , the data handling capacity of the first operator is recorded as M n , the bandwidth utilization of the first operator is denoted as α n , the instruction execution efficiency of the first operator is denoted as β n , the computing power of the second processor is recorded as P n , the bandwidth of the second processor is recorded as B n , then the data transport time t of the first operator compute and t move The calculation method of is as follows.

[0151]

[0152]

[0153] Step 3: Select a larger one from the calculation time of the first operator and the data transfer time of the first operator as the execution time of the first operator.

[0154] In one implementation, the first processor determines the memory occupancy size of the first operator among multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, specifically including S300b.

[0155] S300b. The first processor determines the memory usage size of the second memory during the execution of the first operator according to the dependency relationship of the multiple operators, the execution order of the multiple operators, and the input and output size of the first operator.

[0156] In the embodiment of the present application, the input and output size of the first operator is the memory occupied by the first operator. According to the dependency relationship and execution order of multiple operators, the current memory occupied by the second memory during the operation of the first operator can be determined.

[0157] For example, take two operators among multiple operators as an example, such as the first operator and the second operator. If the first operator and the second operator have a dependency relationship, the execution order is serial execution, for example, the first operator is executed first and then the second operator is executed. If the first operator is currently executed, the memory occupied by the second memory during the operation of the first operator is the memory required to run the first operator. If the first operator and the second operator have no dependency relationship, the execution order is parallel execution, that is, the second operator is also running during the operation of the first operator, then the memory occupied by the second memory during the operation of the first operator is the sum of the memory required to run the first operator and the memory required to run the second operator.

[0158] In one implementation, the first processor determines the memory unloading duration of the first operator among multiple operators of the AI ​​model based on the load information of multiple operators during the AI ​​model training process, specifically including S300c.

[0159] S300c: The first processor determines the memory unloading duration of the first operator according to the memory usage of the first operator, the bandwidth of the second processor, the bandwidth utilization of the second processor, and the latency of the second processor in executing the first operator.

[0160] Optionally, the memory unloading duration of the first operator may be calculated according to the following method:

[0161]

[0162] Among them, T swap represents the memory unloading duration of the first operator, M represents the memory usage of the first operator, N represents the bandwidth of the second processing, R N represents the bandwidth utilization of the second process, T delayIndicates the latency of executing the first operator.

[0163] Optionally, in an embodiment of the present application, the execution time of the above-mentioned operator and the memory occupancy size of the second memory during the operation of the operator can also be obtained from the compiled graph or profiling. The compiled graph or profiling is an information collection tool in the AI ​​model training process.

[0164] It should be understood that the profiling information can only be obtained after the profiling tool is turned on (or logged in), and it takes a certain amount of time to turn on the profiling tool. Based on this, in an embodiment of the present application, an offline knowledge base can be created, and the profiling information of the AI ​​model is stored in the knowledge base. The first processor can query from the knowledge base. If the profiling information of the operator is hit (for example, query by the name of the operator), the above-mentioned operator execution time and the memory occupancy size of the second memory during the operation of the operator are read from the knowledge base; if the profiling information of the operator is not hit in the knowledge base, the operator execution time and the memory size of the second memory during the operation of the operator can be modeled according to the above-mentioned S300a and S300b methods.

[0165] Optionally, when the profiling information of the operator is not found in the knowledge base, the profiling tool can be started to obtain the execution time of the operator and the memory usage of the second memory during the operation of the operator from the latest profiling information online.

[0166] In combination with the above-mentioned S300a, S300b and S300c, the above-mentioned operator execution time, the memory occupancy of the second memory during the operation of the operator and the memory unloading time of the operator are indicators related to the hardware capabilities of the device and the characteristics of the operator, and these indicators are taken into consideration when modeling the optimization strategies for operator recalculation and memory unloading, so as to avoid the problem of low efficiency and high resource consumption in the training process of the AI ​​model due to only selecting a certain optimization strategy based on the theoretical operator calculation amount and memory occupancy.

[0167] It is understandable that, in order to realize the above functions, the above computing device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the method steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0168] The embodiment of the present application can divide the functional modules of the above-mentioned computing device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0169] In the case of dividing each functional module into corresponding functional modules, Figure 5 A possible structural diagram of the computing device involved in the above embodiment is shown. The computing device includes a construction module 501 , a computing module 502 and a sending module 503 .

[0170] The construction module 501 is used to execute S301 in the above method embodiment; the calculation module 502 is used to execute S302 in the above method embodiment; and the sending module 503 is used to execute S303 in the above method embodiment.

[0171] Optionally, the computing device provided in the embodiment of the present application further includes a determination module 504, and the determination module 504 is used to execute S300 (including S300a, S300b, S300c) in the above method embodiment.

[0172] In the case of an integrated unit, Figure 6Another possible structural diagram of the computing device involved in the above embodiment is shown. The computing device may include: a processing module 601 and a communication module 602. The processing module 601 may be used to control and manage the actions of the computing device. For example, the processing module 601 may be used to support the computing device to execute S300, S301, S302 in the above method embodiment, and / or other processes for the technology described herein. The communication module 602 may be used to support the computing device to communicate with other network entities. For example, the communication module 602 may be used to support the computing device to execute S303 in the above method embodiment. Optionally, as Figure 6 As shown, the computing device may further include a storage module 603 for storing program codes and data of the computing device.

[0173] The processing module 601 may be a processor, for example, the processor may be Figure 2 The communication module 602 may be a transceiver, a transceiver circuit or a communication interface, for example Figure 2 The communication interface 203 in the storage module 603 may be a memory, for example Figure 2 Memory 202 in.

[0174] The various modules of the above-mentioned computing device can also be used to perform other actions in the above-mentioned method embodiment. All relevant contents of each step involved in the above-mentioned method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0175] For more details on how the modules included in the computing device implement the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here. Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments.

[0176] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) mode or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a magnetic disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state drive (SSD)).

[0177] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk, and other media that can store program codes.

[0182] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating an AI model, characterized in that: Applied to a computing device including a first processor and a second processor, the memory of the first processor is a first memory, the memory of the second processor is a second memory, the first processor is used to control the second processor so that the second processor executes multiple operators of an AI model to train the AI ​​model, and the method includes: The first processor constructs a target optimization function with the operation instruction information as the optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator; wherein the memory unloading time is the time taken by the second processor to obtain data required to execute the first operator from the first memory when the data in the second memory is unloaded to the first memory, and the first operator is any one of the multiple operators; Performing global optimization on the target optimization function to obtain the operation indication information; the operation indication information is used to indicate whether to perform a target operation on each of the multiple operators, the target operation including a recalculation operation or a memory unloading operation; The first processor sends an adjustment instruction to the second processor; the adjustment instruction includes the operation instruction information, and the adjustment instruction is used to instruct the second processor to execute the multiple operators according to the operation instruction information to train the AI ​​model to obtain a target AI model.

2. The method according to claim 1, characterized in that The target optimization function takes the execution time and memory unloading time of the operator as optimization targets, and takes the memory occupancy size of the second memory during the operation of the operator as a constraint condition.

3. The method according to claim 2, characterized in that The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i represents the memory usage of the second memory during the operation of the i-th operator, E total Indicates the total memory usage for training the AI ​​model, E max Indicates the constraint value of available memory; x i It is used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

4. The method according to claim 1, characterized in that The target optimization function takes the memory usage of the second memory during the operation of the operator as the optimization target, and takes the execution time of the operator and the memory unloading time as the constraint conditions.

5. The method according to claim 4, characterized in that The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i represents the memory usage of the second memory during the operation of the i-th operator, T target Indicates the total time of training the AI ​​model; x i It is used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

6. The method according to claim 1, characterized in that The target optimization function takes the execution time of the operator, the memory unloading time and the memory occupancy size of the second memory during the operation of the operator as optimization targets, and takes the execution time of the operator, the memory unloading time and the memory occupancy size as constraints.

7. The method according to claim 6, characterized in that The AI ​​model includes n operators, where n is an integer greater than or equal to 2; The objective optimization function is: Among them, T i represents the execution time of the ith operator, T swap_i represents the memory unloading duration of the i-th operator, E i represents the memory usage of the second memory during the operation of the i-th operator, T target Indicates the total time of training the AI ​​model, E total Indicates the total memory usage for training the AI ​​model, E max Represents the constraint value of available memory; α, β are weight coefficients; x i It is used to indicate whether to perform the target operation on the i-th operator, where x i = 1, the target operation is performed on the i-th operator. When x i =0, the target operation is not performed on the i-th operator.

8. The method according to any one of claims 3 to 7, characterized in that: When x i =1, and min(T i ,T swap_i )=T i , performing the recalculation operation on the i-th operator; When x i =1, and min(T i ,T swap_i )=T swap_i , performing the memory unloading operation on the i-th operator.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: The first processor determines the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator based on the load information of the multiple operators during the training process of the AI ​​model.

10. The method according to claim 9, characterized in that The load information of the multiple operators includes memory access information of each operator in the multiple operators and / or a cache hit rate of each operator; The first processor determines, according to load information of the multiple operators in the AI ​​model training process, an execution duration of a first operator among multiple operators of the AI ​​model, including: The first processor determines the execution time of the first operator according to memory access information of the first operator, a cache hit rate of the first operator, a calculation amount of the first operator, a bandwidth utilization rate of the first operator, an instruction execution efficiency of the first operator, a computing power of the second processor, and a bandwidth of the second processor.

11. The method according to claim 10, characterized in that The load information of the multiple operators also includes at least one of the following: dependencies of the multiple operators, execution order of the multiple operators, or input and output size of each of the multiple operators; The first processor determines, according to load information of the multiple operators during the training process of the AI ​​model, a memory occupancy size of the second memory during the operation of a first operator among the multiple operators of the AI ​​model, including: The first processor determines the memory occupancy size of the second memory during the execution of the first operator according to the dependency relationship of the multiple operators, the execution order of the multiple operators, and the input and output size of the first operator.

12. The method according to claim 11, characterized in that The load information of the multiple operators also includes the memory usage of each of the multiple operators; The first processor determines, according to load information of the multiple operators during the training process of the AI ​​model, a memory unloading duration of a first operator among multiple operators of the AI ​​model, including: The first processor determines a memory unloading duration of the first operator according to a memory occupation size of the first operator, a bandwidth of the second processor, a bandwidth utilization rate of the second processor, and a latency of the second processor executing the first operator.

13. A computing device, characterized in that: It includes a building module, a computing module and a sending module; The construction module is used to construct a target optimization function with operation instruction information as the optimization variable based on the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator; wherein the second memory is the memory of the second processor, the memory unloading time is the time for the second processor to obtain data required for executing the first operator from the first memory when the data in the second memory is unloaded to the first memory, the first operator is any one of the multiple operators, and the first memory is the memory of the first processor; The computing module is used to perform global optimization on the target optimization function to obtain the operation indication information; the operation indication information is used to indicate whether to perform a target operation on each of the multiple operators, and the target operation includes a recalculation operation or a memory unloading operation; The sending module is used to send an adjustment instruction to the second processor; the adjustment instruction includes the operation instruction information, and the adjustment instruction is used to instruct the second processor to execute the multiple operators according to the operation instruction information to train the AI ​​model to obtain a target AI model.

14. The computing device according to claim 13, characterized in that The computing device further includes a determination module; The determination module is used to determine the execution time of each of the multiple operators of the AI ​​model, the memory occupancy of the second memory during the operation of each operator, and the memory unloading time of each operator according to the load information of the multiple operators during the training process of the AI ​​model.

15. The computing device according to claim 14, characterized in that The load information of the multiple operators includes memory access information of each operator in the multiple operators and / or a cache hit rate of each operator; The determination module is specifically used to determine the execution time of the first operator according to the memory access information of the first operator, the cache hit rate of the first operator, the calculation amount of the first operator, the bandwidth utilization of the first operator, the instruction execution efficiency of the first operator, the computing power of the second processor, and the bandwidth of the second processor.

16. The computing device according to claim 14, characterized in that The load information of the multiple operators also includes at least one of the following: dependencies of the multiple operators, execution order of the multiple operators, or input and output size of each of the multiple operators; The determination module is specifically used to determine the memory usage size of the second memory during the operation of the first operator according to the dependency relationship of the multiple operators, the execution order of the multiple operators, and the input and output size of the first operator.

17. The computing device according to claim 14, characterized in that The load information of the multiple operators also includes the memory usage of each of the multiple operators; The determination module is specifically used to determine the memory unloading duration of the first operator according to the memory usage size of the first operator, the bandwidth of the second processor, the bandwidth utilization rate of the second processor, and the latency of the second processor executing the first operator.

18. A computing device, characterized in that It includes a memory and at least one processor connected to the memory, the at least one processor includes a first processor and a second processor; the memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the at least one processor, the first processor executes the method as described in any one of claims 1 to 12, and the second processor executes multiple operators of the AI ​​model to train the AI ​​model.

19. A computer-readable storage medium, characterized in that: Computer instructions are stored, and when the computer instructions are executed on a computer, the method according to any one of claims 1 to 12 is executed.

Images