GPU-oriented Artificial Intelligence Model Deployment Method and Device

Abstract

The present invention provides a method and device for deploying an artificial intelligence model for a GPU, including: determining the types of operators supported by the NPU built into the GPU, converting the front-end model into an expected language model using a preset compilation framework, and locating the operators in the expected language model; determining that the operators of the expected language model belong to the types of operators, removing predetermined layers from the expected language model, combining predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combining the convolutional layer with the batch normalization layer to obtain a second separate convolutional layer; obtaining the input data size and output data size of each layer in the expected language model, and determining the memory size allocated during the operation of the expected language model according to their sizes; using a GPU compiler and an NPU compiler to perform layer-by-layer compilation on the expected language model and scheduling the execution of the compilation results to complete model deployment. The present invention is beneficial to the generalization deployment of artificial intelligence models.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a method and device for deploying an artificial intelligence model for a GPU. Background Art

[0002] Currently, with the acceleration of the process of domestic GPU replacement and the further development of the AI (Artificial Intelligence) computing field, there is a wide demand for using domestic graphics cards for AI operations. On the one hand, there are numerous AI operation operators and the graph structure of AI models is complex. On the other hand, there are huge differences between different hardware structures, and it is difficult to deploy various deep learning models on different chips. Correspondingly, there is a wide demand for AI model applications on some domestic GPU chips. Therefore, developing a method and device for deploying an artificial intelligence model for a GPU to effectively overcome the defects in the above related technologies has become a technical problem urgently to be solved in the industry. Summary of the Invention

[0003] In view of the above problems existing in the prior art, embodiments of the present invention provide a method and device for deploying an artificial intelligence model for a GPU.

[0004] In a first aspect, an embodiment of the present invention provides a method for deploying an artificial intelligence model for a GPU, including: determining the operator types supported by the NPU built in the GPU, converting the front-end model into a desired language model using a preset compilation framework, and locating the operators in the desired language model; determining that the operators in the desired language model belong to the operator types, removing predetermined layers in the desired language model, combining predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combining the convolutional layer with the batch normalization layer to obtain a second separate convolutional layer; obtaining the input data size and output data size of each layer in the desired language model, and determining the memory size allocated during the operation of the desired language model according to the sizes of the input data size and the output data size; performing layer-by-layer compilation on the desired language model using a GPU compiler and an NPU compiler, and scheduling and executing the compilation result using a preset compilation framework to complete model deployment.

[0005] Based on the content of the above method embodiment, in the method for deploying an artificial intelligence model for a GPU provided in an embodiment of the present invention, the locating of the operators in the desired language model includes: determining that the operation positions of the operators in the desired language model are in the GPU or the NPU, and recording the operation positions in a map.json file; wherein, the desired language model is a RelayIR language model.

[0006] Based on the content of the above method embodiments, in the artificial intelligence model deployment method for GPU provided in the embodiments of the present invention, determining that the operator of the desired language model belongs to the operator type includes: traversing the operators in the desired language model layer by layer, and determining whether the operators in one layer of the desired language model belong to a convolution operator, a max pooling operator, a subtraction operator, an addition operator, a multiplication operator, or a division operator; if the determination result is yes, add the operation device of the operators in this layer as Device_NPU in the map.json file and return; if the determination result is no, add the operation device of the operators in this layer as Device_GPU in the map.json file and return.

[0007] Based on the content of the above method embodiments, in the artificial intelligence model deployment method for GPU provided in the embodiments of the present invention, combining the predetermined parameter with the bias constant of the convolutional layer to obtain the first separate convolutional layer includes: traversing the operators of the desired language model, determining whether the operator is a convolutional operator, if it is not a convolutional operator, repeat the determination for the next operator until all operators are traversed, if it is a convolutional operator, determine whether the next operator of this convolutional operator is an addition operator or a subtraction operator, if not, return, if so, determine whether one of the two inputs of this addition operator or subtraction operator is the output of this convolutional operator and the other is a constant, and the data size of the constant is the same as the data size of the bias constant of this convolutional operator or the data size of the constant is 1, if not, return, if so, if it is an addition operator, take out the constant and add the bias constant in the convolutional layer, update the bias constant of this convolutional operator, and remove this addition operator; if it is a subtraction operator, subtract the constant from the bias constant in the convolutional layer, update the bias constant of this convolutional operator, and remove this subtraction operator.

[0008] Based on the content of the above method embodiments, in the artificial intelligence model deployment method for GPU provided in the embodiments of the present invention, combining the convolutional layer with the batch normalization layer to obtain the second separate convolutional layer includes: traversing the operators of the desired language model, determining whether the operator is a convolutional operator, if not, repeat the determination for the next operator until all operators are traversed, if it is a convolutional operator, determine whether the next operator of this convolutional operator is a batch normalization operator, if not, return, if so, obtain the weight and bias constant of the convolution kernel of this convolutional operator, obtain the parameters of the batch normalization operator, change the weight and bias constant of the convolution kernel of this convolutional operator, remove this batch normalization operator and return.

[0009] Based on the content of the above method embodiments, in the artificial intelligence model deployment method for GPU provided in the embodiments of the present invention, the obtaining of the input data size and output data size of each layer in the expected language model includes: calculating the output data size based on the input data size, calculating the output data size of the first layer. The input data size of the first layer is the input data size of the entire model, so the input data size of the first layer is fixed. Calculate the output data size. If the output data size can be calculated, then start traversing each layer of operators from the second layer. The output data size of the upper layer operator is the input data size of the current operator, and calculate the output data size. If the data size can be calculated, then return. Otherwise, calculate the data size of the corresponding layer in the front-end inference framework, record it in the expected language model and return; after calculating the data size of each layer of the expected language model, use the front-end framework to calculate the input and output front-end data sizes of all layers of the corresponding front-end model, and check whether the front-end data size is equal to all the corresponding data sizes of the expected language model. If not, replace the inference data size in the expected language model with the data size corresponding to the front-end data size.

[0010] Based on the content of the above method embodiments, in the artificial intelligence model deployment method for GPU provided in the embodiments of the present invention, the step of performing layer-by-layer compilation on the expected language model using a GPU compiler and an NPU compiler and scheduling and executing the compilation results using a preset compilation framework to complete model deployment includes: for the operators marked as Device_NPU executable, map them to the first binary machine code for operation in the NPU through the NPU compiler; for the operators marked as Device_GPU executable, map them to the second binary machine code for operation in the GPU through the GPU compiler; use the TVM compilation framework to schedule and execute the first binary machine code and the second binary machine code to complete model deployment.

[0011] In a second aspect, an embodiment of the present invention provides an artificial intelligence model deployment device for a GPU, including: a first main module, configured to determine the operator types supported by the NPU built in the GPU, convert the front-end model into a desired language model using a preset compilation framework, and locate the operators in the desired language model; a second main module, configured to determine that the operators of the desired language model belong to the operator types, remove predetermined layers from the desired language model, combine predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combine the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer; a third main module, configured to obtain the input data size and output data size of each layer in the desired language model, and determine the memory size allocated during the operation of the desired language model according to the sizes of the input data size and the output data size; a fourth main module, configured to perform layer-by-layer compilation on the desired language model using a GPU compiler and an NPU compiler, and schedule the execution of the compilation results using a preset compilation framework to complete model deployment.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0013] at least one processor; and

[0014] at least one memory communicatively connected to the processor, wherein:

[0015] The memory stores program instructions executable by the processor, and the processor can execute the artificial intelligence model deployment method provided by any one of the various implementation manners in the first aspect by invoking the program instructions.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the artificial intelligence model deployment method provided by any one of the various implementation manners in the first aspect.

[0017] The artificial intelligence model deployment method and device provided by the embodiments of the present invention can efficiently deploy the artificial intelligence model on domestic GPU chips by using the TVM open-source compilation framework, and have strong compatibility with the types of artificial intelligence models, which is beneficial to the generalization deployment of artificial intelligence models. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 Flowchart of the artificial intelligence model deployment method for GPU provided by an embodiment of the present invention;

[0020] Figure 2 Schematic structural diagram of the artificial intelligence model deployment device for GPU provided by an embodiment of the present invention;

[0021] Figure 3 Schematic physical structure diagram of an electronic device provided by an embodiment of the present invention. Specific implementation manners

[0022] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. In addition, the technical features in each embodiment or a single embodiment provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the order of steps and / or the pattern of structural composition, but must be based on the fact that those of ordinary skill in the art can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0023] The main objective of the present invention is to provide a set of AI model compilation and deployment methods based on TVM (an open-source, end-to-end deep learning model compilation framework), and this method can also be extended to other chips. Based on this idea, an embodiment of the present invention provides an artificial intelligence model deployment method for GPU. Refer to Figure 1 , this method includes: determining the operator types supported by the NPU built in the GPU, converting the front-end model into a desired language model using a preset compilation framework, and locating the operators in the desired language model; determining that the operators in the desired language model belong to the operator types, removing the predetermined layers in the desired language model, combining the predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combining the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer; obtaining the input data size and output data size of each layer in the desired language model, and determining the memory size allocated during the operation of the desired language model according to the size of the input data size and the size of the output data size; layer-by-layer compiling the desired language model using a GPU compiler and an NPU compiler, and scheduling and executing the compilation result using a preset compilation framework to complete model deployment. It can be understood that the following takes the "FengHua No. 1" graphics card of Innosilicon as an example for illustration, but the present invention is not limited thereto.

[0024] In another embodiment, the operator types supported by the GPU with built-in NPU need to pre-define that the operator types supported by the built-in NPU hardware of "FengHua No. 1" are "Conv" (convolution), "Maxpooling" (max pooling), and "binary_op" (addition, subtraction, multiplication, and division). The preset compilation framework can adopt the TVM compilation framework. TVM is an open-source, end-to-end deep learning model compilation framework used to optimize the inference running speed of deep learning models in any target environment such as CPUs, GPUs, and ARMs. Common application scenarios include: scenarios that need to be compatible with all mainstream models as inputs and generate optimized deployment models for any type of target hardware; scenarios with strict requirements for performance metrics such as inference latency and throughput of the deployment model; scenarios that need to customize model operators, develop self-owned target hardware, and customize the model optimization process. Usually, the models exported by deep learning frameworks (TensorFlow, PyTorch, MXNet, etc.) are used as the inputs of the TVM compilation framework. After the graph optimization operations and operator-level automatic optimization operations within the framework, they are finally converted into deployment models for the target runtime (CPU / GPU / ARM, etc.). The optimized models can theoretically maximize the utilization of the resources of the target hardware to minimize the inference latency of the models.

[0025] Based on the content of the above method embodiments, as an optional embodiment, in the method for deploying an artificial intelligence model for a GPU provided in the embodiments of the present invention, the positioning of the operators in the expected language model includes: determining that the operation positions of the operators in the expected language model are in the GPU or the NPU, and recording the operation positions in the map.json file; wherein, the expected language model is a RelayIR language model.

[0026] In another embodiment, the front-end framework model (.onnx,.pb,.pt) is converted into a RelayIR language model (i.e., the expected language model) through TVM, and a map.json file in json format is preset. Among them, the RelayIR language is a versatile programming language used for the intermediate representation of machine learning systems. The RelayIR language supports algebraic data types, closures, control flow, and recursion, so that it can directly represent models more complex than the IR based on computational graphs. The RelayIR language also includes a form of dependent type using type relations to handle the operator shape analysis with complex requirements for parameter shapes. The RelayIR language is designed to be extensible, making it easy to develop new large-scale program transformations and optimizations for machine learning development.

[0027] Based on the content of the above method embodiments, as an alternative embodiment, in the method for deploying an artificial intelligence model for GPU provided in the embodiments of the present invention, determining that the operator of the desired language model belongs to the operator type includes: traversing the operators in the RelayIR language model layer by layer, and determining whether the operator in one layer of the RelayIR language model belongs to a convolution operator, a max pooling operator, a subtraction operator, an addition operator, a multiplication operator, or a division operator; if the determination result is yes, add the computing device of the operator in this layer to the map.json file as Device_NPU and return; if the determination result is no, add the computing device of the operator in this layer to the map.json file as Device_GPU and return.

[0028] In another embodiment, determining that the operator of the desired language model belongs to the operator type includes: Step 1, traverse the RelayIR operators layer by layer, and determine whether the operator belongs to one of {"Conv", "Maxpooling", "Sub", "Add", "Mul", "Div"}. If it is, execute Step 2; if not, execute Step 3; Step 2, add the computing device of the operator in this layer to the map.json as "Device_NPU" and return to Step 1; Step 3, add the computing device of the operator in this layer to the map.json as "Device_GPU" and return to Step 1.

[0029] Based on the content of the above method embodiments, as an alternative embodiment, in the method for deploying an artificial intelligence model for GPU provided in the embodiments of the present invention, merging the predetermined parameter with the bias constant of the convolutional layer to obtain a first separate convolutional layer includes: traversing the operators of the RelayIR language model, and determining whether the operator is a convolutional operator. If it is not a convolutional operator, repeat the determination for the next operator until all operators are traversed. If it is a convolutional operator, determine whether the next operator of the convolutional operator is an addition operator or a subtraction operator. If not, return. If so, determine whether one of the two inputs of the addition operator or the subtraction operator is the output of the convolutional operator, and the other is a constant, and the data size of the constant is the same as the data size of the bias constant of the convolutional operator or the data size of the constant is 1. If not, return. If so, if it is an addition operator, take out the constant and add the bias constant in the convolutional layer, update the bias constant of the convolutional operator, and remove the addition operator; if it is a subtraction operator, subtract the constant from the bias constant in the convolutional layer, update the bias constant of the convolutional operator, and remove the subtraction operator.

[0030] In another embodiment, the Dropout layer and the Flatten layer in the RelayIR language model are removed, and then the following steps are performed: 1. Traverse the model in RelayIR and determine whether the model is a convolutional operator of the "Conv" type. If not, repeat the determination for the next operator until all operators are traversed. If it is a "Conv" type operator, perform step 2. 2. Determine whether the next operator of the Conv operator is an Add operator or a Sub operator in "binary_op". If it is an add or sub operator in "binary_op", perform step 3. If not, return to step 1. 3. Determine whether the two inputs of the "binary op" operator are one the output of the Conv operator and the other an AI model constant constant, and the shape of constant is the same as the shape of the bias constant of the Conv layer or the shape of constant is [1]. If so, perform step 4. If not, return to step 1. 4. If it is an Add operator (addition operator), take out the constant constant, then take out the bias (i.e., the bias constant) in the Conv layer, bias = bias + constant, update the new bias of the Conv, and remove the add operator. If it is a Sub operator (subtraction operator), then bias = bias - constant, update the new bias of the Conv, and remove the sub operator; perform step 5. 5. Determine whether the next operator is still an Add operator or a Sub operator in the "binary_op" operator. If it is this operator, perform step 3. If not, perform step 1.

[0031] Based on the content of the above method embodiment, as an optional embodiment, in the method for deploying an artificial intelligence model for GPU provided in the embodiment of the present invention, the combining the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer includes: traversing the operators of the RelayIR language model, determining whether the operator is a convolutional operator. If not, repeat the determination for the next operator until all operators are traversed. If it is a convolutional operator, determine whether the next operator of the convolutional operator is a batch normalization operator. If not, return. If so, obtain the weights and bias constants of the convolution kernel of the convolutional operator, obtain the parameters of the batch normalization operator, change the weights and bias constants of the convolution kernel of the convolutional operator, remove the batch normalization operator and return.

[0032] In another embodiment, 1. Traverse the operators in the RelayIR language model to determine whether the model is a convolutional operator of the "Conv" type. If not, repeat the determination for the next operator until all operators are traversed. If it is a "Conv" type operator, perform step 2. 2. Determine whether the next operator of the Conv operator is "Batch_normalization" (i.e., batch normalization). If it is a Batch_normalization operator, perform step 3. If it is not Batch_normalization, return to step 1 to continue traversing. 3. Obtain the weight and bias constant parameters weight and bias of the convolution kernel of the convolutional layer, and obtain the gamma, beta, mean, var, and eps parameters of the Batch_normalization layer. Then B = beta - mean * M, weight = weight * M, bias = bias + B. Where mean represents the mean, var represents the variance, gamma represents the first parameter factor, beta represents the second parameter factor, eps represents the third parameter factor, B represents the first intermediate variable, and M represents the second intermediate variable. Modify the weight parameter and bias parameter in the convolutional layer, remove the Batch_normalization layer at the same time, and return to step 1.

[0033] Based on the content of the above method embodiment, as an optional embodiment, in the method for deploying an artificial intelligence model for GPU provided in the embodiment of the present invention, the obtaining the input data size and output data size of each layer in the desired language model includes: calculating the output data size based on the input data size, calculating the output data size of the first layer. The input data size of the first layer is the input data size of the entire model, so the input data size of the first layer is fixed. Calculate the output data size. If the output data size can be calculated, start traversing each layer of operators from the second layer. The output data size of the upper layer operator is the input data size of the current operator, and calculate the output data size. If the data size can be calculated, return. Otherwise, calculate the data size of the corresponding layer in the front-end inference framework, record it in the RelayIR language model and return; after calculating the data size of each layer of the RelayIR language model, use the front-end framework to calculate the input and output front-end data sizes of all layers of the corresponding front-end model, and check whether the front-end data size is equal to all the corresponding data sizes in the RelayIR language model. If not, replace the inference data size in the RelayIR language model with the data size corresponding to the front-end data size.

[0034] In another embodiment, obtaining the input data size and output data size of each layer in the desired language model includes: 1. Pre-defining the operator shape (i.e., data size, the same below) inference method. Since the calculation method of the operator for each layer is fixed, the output shape can be inferred based on the input. 2. Inferring the output shape of the first layer. Since the input of the first layer of the model is the input of the entire model, its input shape is fixedly known. According to the output shape inference predetermined in step 1, if the shape can be inferred in step 1, then execute step 3; otherwise, execute step 4. 3. Starting from the second layer of the model, traverse each layer of the operator. Among them, the output shape of the upper-layer operator is the input shape of the current operator. According to the output shape inference predetermined in step 1, if the shape can be inferred in step 1, then return to execute step 3; otherwise, execute step 4. 4. Infer the shape of the corresponding layer in the front-end inference framework (such as torch, tensorflow, onnx), and record it in the RelayIR language model, then return to step 3. 5. After inferring the shape of each layer of the RelayIR language model, use the front-end framework (torch, tensorflow, onnx) to infer the input and output front-shapes (i.e., front-end data sizes) of all layers of the corresponding front-end model, and check whether the front-shape is equal to all the corresponding shapes of the layers of the RelayIR language model. If they are not equal, then replace the inferred shape in RelayIR with the shape corresponding to the front-shape.

[0035] Based on the content of the above method embodiments, as an optional embodiment, in the method for deploying an artificial intelligence model for GPU provided in the embodiments of the present invention, the desired language model is compiled layer by layer using a GPU compiler and an NPU compiler, and the preset compilation framework is used to schedule and execute the compilation results to complete model deployment, including: Among them, the operator marked as Device_NPU executable is mapped to the first binary machine code for operation in the NPU through the NPU compiler; the operator marked as Device_GPU executable is mapped to the second binary machine code for operation in the GPU through the GPU compiler; the TVM compilation framework is used to schedule and execute the first binary machine code and the second binary machine code to complete model deployment.

[0036] In another embodiment, the operators marked as "Device_NPU" executable are directly mapped to the binary machine code executable inside the "FengHua No.1" NPU through the "FengHua No.1" NPU compiler. For the operators marked as "Device_GPU" executable, the "FengHua No.1" GPU compiler is used to generate the binary machine code that can be operated in the GPU. Finally, the TVM compilation framework is used to coordinate and schedule the execution of the two binary machine codes to complete the final deployment and operation.

[0037] The method for deploying an artificial intelligence model for a GPU provided by the embodiments of the present invention can efficiently deploy the artificial intelligence model on a domestic GPU chip by using the TVM open-source compilation framework, and has strong compatibility with the types of artificial intelligence models, which is beneficial to the generalization deployment of artificial intelligence models.

[0038] The implementation basis of each embodiment of the present invention is achieved through programmed processing by a device with processor functions. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention can be encapsulated into various modules. Based on this actual situation, on the basis of the above embodiments, the embodiments of the present invention provide a device for deploying an artificial intelligence model for a GPU, and this device is used to execute the method for deploying an artificial intelligence model for a GPU in the above method embodiments. See Figure 2 , the device includes: a first main module, which is used to determine the type of operators supported by the NPU built in the GPU, convert the front-end model into an expected language model by using a preset compilation framework, and locate the operators in the expected language model; a second main module, which is used to determine that the operators of the expected language model belong to the operator type, remove the predetermined layers in the expected language model, combine the predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combine the convolutional layer with the batch normalization layer to obtain a second separate convolutional layer; a third main module, which is used to obtain the input data size and output data size of each layer in the expected language model, and determine the memory size allocated during the operation of the expected language model according to the size of the input data size and the size of the output data size; a fourth main module, which is used to compile the expected language model layer by layer by using a GPU compiler and an NPU compiler, and schedule and execute the compilation results by using a preset compilation framework to complete the model deployment.

[0039] The device for deploying an artificial intelligence model for a GPU provided by the embodiments of the present invention adopts Figure 2 several modules in it. By using the TVM open-source compilation framework, the artificial intelligence model can be efficiently deployed on a domestic GPU chip, and has strong compatibility with the types of artificial intelligence models, which is beneficial to the generalization deployment of artificial intelligence models.

[0040] It should be noted that the device in the device embodiment provided by the present invention can be used not only to implement the method in the above method embodiment, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in setting corresponding functional modules, and its principle is basically the same as that of the above device embodiment provided by the present invention. As long as those skilled in the art, on the basis of the above device embodiment, refer to the specific technical solutions in other method embodiments, obtain corresponding technical means by combining technical features, and the technical solutions constituted by these technical means, and on the premise of ensuring the practicability of the technical solutions, the device in the above device embodiment can be improved to obtain corresponding device type embodiments for implementing the methods in other method type embodiments. For example:

[0041] Based on the content of the above device embodiment, as an optional embodiment, the artificial intelligence model deployment device for GPU provided in the embodiment of the present invention further includes: a first sub-module, configured to implement the positioning of the operator in the expected language model, including: determining that the operation position of the operator in the expected language model is in the GPU or the NPU, and recording the operation position in the map.json file; wherein, the expected language model is the RelayIR language model.

[0042] Based on the content of the above device embodiment, as an optional embodiment, the artificial intelligence model deployment device for GPU provided in the embodiment of the present invention further includes: a second sub-module, configured to implement the determination that the operator of the expected language model belongs to the operator type, including: traversing the operators in the RelayIR language model layer by layer, and determining whether the operator in one layer of the RelayIR language model belongs to a convolutional operator, a max pooling operator, a subtraction operator, an addition operator, a multiplication operator, or a division operator; if the determination result is yes, add the operation device of the operator in this layer as Device_NPU in the map.json file and return; if the determination result is no, add the operation device of the operator in this layer as Device_GPU in the map.json file and return.

[0043] Based on the content of the above device embodiments, as an alternative embodiment, the AI model deployment device for GPU provided in the embodiments of the present invention further includes: a third sub-module for implementing the combination of the predetermined parameters and the bias constant of the convolutional layer to obtain a first separate convolutional layer, including: traversing the operators of the RelayIR language model, determining whether the operator is a convolutional operator, if it is not a convolutional operator, repeating the determination for the next operator until all operators are traversed, if it is a convolutional operator, determining whether the next operator of the convolutional operator is an addition operator or a subtraction operator, if it is not, returning, if it is, determining whether one of the two inputs of the addition operator or the subtraction operator is the output of the convolutional operator and the other is a constant, and the data size of the constant is the same as the data size of the bias constant of the convolutional operator or the data size of the constant is 1, if it is not, returning, if it is, if it is an addition operator, taking out the constant and adding the bias constant in the convolutional layer, updating the bias constant of the convolutional operator, and removing the addition operator; if it is a subtraction operator, subtracting the constant from the bias constant in the convolutional layer, updating the bias constant of the convolutional operator, and removing the subtraction operator.

[0044] Based on the content of the above device embodiments, as an alternative embodiment, the AI model deployment device for GPU provided in the embodiments of the present invention further includes: a fourth sub-module for implementing the combination of the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer, including: traversing the operators of the RelayIR language model, determining whether the operator is a convolutional operator, if it is not, repeating the determination for the next operator until all operators are traversed, if it is a convolutional operator, determining whether the next operator of the convolutional operator is a batch normalization operator, if it is not, returning, if it is, obtaining the weight and bias constant of the convolutional kernel of the convolutional operator, obtaining the parameters of the batch normalization operator, changing the weight and bias constant of the convolutional kernel of the convolutional operator, removing the batch normalization operator and returning.

[0045] Based on the content of the above device embodiments, as an alternative embodiment, the AI model deployment device for GPU provided in the embodiments of the present invention further includes: a fifth sub-module, configured to implement obtaining the input data size and output data size of each layer in the desired language model, including: calculating the output data size based on the input data size, calculating the output data size of the first layer, where the input data size of the first layer is the input data size of the entire model, so the input data size of the first layer is fixed, calculating the output data size, if the output data size can be calculated, then traverse each layer of operators starting from the second layer, where the output data size of the upper-layer operator is the input data size of the current operator, and calculate the output data size, if the data size can be calculated, then return, otherwise infer the data size of the corresponding layer in the front-end inference framework, record it in the RelayIR language model and return; after calculating the data size of each layer of the RelayIR language model, use the front-end framework to infer the input and output front-end data sizes of all layers of the corresponding front-end model, and check whether the front-end data size is equal to all corresponding data sizes of the RelayIR language model, if not, then replace the inference data size in the RelayIR language model with the data size corresponding to the front-end data size.

[0046] Based on the content of the above device embodiments, as an alternative embodiment, the AI model deployment device for GPU provided in the embodiments of the present invention further includes: a sixth sub-module, configured to implement layer-by-layer compilation of the desired language model using a GPU compiler and an NPU compiler, and use a preset compilation framework to schedule and execute the compilation results to complete model deployment, including: for operators marked as Device_NPU executable, map them to the first binary machine code for operation in the NPU through the NPU compiler; for operators marked as Device_GPU executable, map them to the second binary machine code for operation in the GPU through the GPU compiler; use the TVM compilation framework to schedule and execute the first binary machine code and the second binary machine code to complete model deployment.

[0047] The method of the embodiments of the present invention is implemented relying on an electronic device, so it is necessary to introduce the relevant electronic device. For this purpose, the embodiments of the present invention provide an electronic device, such as Figure 3 shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, where at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. The at least one processor can call the logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0048] In addition, when the logic instructions in at least one of the above-mentioned memories are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various method embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0049] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0050] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0051] The flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. Based on this understanding, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or sometimes in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0052] It should be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, the elements defined by the phrase "comprise..." do not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the elements.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence model deployment method for GPUs, characterized in that, Including: Determine the operator types supported by the NPU built into the GPU, convert the front-end model into an expected language model using a preset compilation framework, and locate the operators in the expected language model; Determine whether the operators in the expected language model belong to the operator types, remove the predetermined layers in the expected language model, combine the predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combine the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer; wherein, the determining whether the operators in the expected language model belong to the operator types includes: traversing the operators in the expected language model layer by layer, and determining whether the operators in one layer of the expected language model belong to convolutional operators, max pooling operators, subtraction operators, addition operators, multiplication operators, or division operators; if the judgment result is yes, add the operation device of the operators in the layer to Device_NPU in the map.json file and return; if the judgment result is no, add the operation device of the operators in the layer to Device_GPU in the map.json file and return; Obtain the input data size and output data size of each layer in the expected language model, and determine the memory size allocated during the operation of the expected language model according to the size of the input data size and the size of the output data size; Perform layer-by-layer compilation on the expected language model using a GPU compiler and an NPU compiler, and use a preset compilation framework to schedule the execution of the compilation results to complete model deployment, including: for operators marked as executable by Device_NPU, map them to the first binary machine code for operation in the NPU through the NPU compiler; for operators marked as executable by Device_GPU, map them to the second binary machine code for operation in the GPU through the GPU compiler; use the TVM compilation framework to schedule the execution of the first binary machine code and the second binary machine code to complete model deployment.

2. The method for deploying an artificial intelligence model for a GPU according to claim 1, wherein The locating the operators in the expected language model includes: determining that the operation location of the operators in the expected language model is in the GPU or the NPU, and recording the operation location in the map.json file, wherein the expected language model includes a RelayIR language model.

3. The method for deploying an artificial intelligence model for a GPU according to claim 2, wherein The merging of the predetermined parameter with the bias constant of the convolutional layer to obtain the first separate convolutional layer includes: traversing the operators of the desired language model, determining whether the operator is a convolutional operator. If it is not a convolutional operator, repeat the determination for the next operator until all operators are traversed. If it is a convolutional operator, determine whether the next operator of the convolutional operator is an addition operator or a subtraction operator. If it is not, return. If it is, determine whether one of the two inputs of the addition operator or subtraction operator is the output of the convolutional operator and the other is a constant, and the data size of the constant is the same as the data size of the bias constant of the convolutional operator or the data size of the constant is 1. If it is not, return. If it is, if it is an addition operator, take out the constant, add it to the bias constant taken out from the convolutional layer, update the bias constant of the convolutional operator, and remove the addition operator; if it is a subtraction operator, subtract the constant from the bias constant in the convolutional layer, update the bias constant of the convolutional operator, and remove the subtraction operator.

4. The method for deploying an artificial intelligence model oriented to a GPU according to claim 2, wherein The combination of the convolutional layer and the batch normalization layer to obtain the second separate convolutional layer includes: traversing the operators of the desired language model, determining whether the operator is a convolutional operator. If it is not, repeat the determination for the next operator until all operators are traversed. If it is a convolutional operator, determine whether the next operator of the convolutional operator is a batch normalization operator. If it is not, return. If it is, obtain the weights and bias constants of the convolutional kernel of the convolutional operator, obtain the parameters of the batch normalization operator, change the weights and bias constants of the convolutional kernel of the convolutional operator, remove the batch normalization operator and return.

5. The method for deploying an artificial intelligence model for a GPU according to claim 2, characterized in that, The obtaining of the input data size and output data size of each layer in the desired language model includes: calculating the output data size based on the input data size, calculating the output data size of the first layer. The input data size of the first layer is the input data size of the entire model, so the input data size of the first layer is fixed. Calculate the output data size. If the output data size can be calculated, start traversing each layer of operators from the second layer. The output data size of the upper-layer operator is the input data size of the current operator, and calculate the output data size. If the data size can be calculated, return. Otherwise, calculate the data size of the corresponding layer in the front-end inference framework, record it in the desired language model and return; after calculating the data sizes of each layer of the desired language model, use the front-end framework to calculate the input and output front-end data sizes of all layers of the corresponding front-end model, and check whether the front-end data sizes are equal to the corresponding data sizes of the desired language model. If they are not equal, replace the inference data sizes in the desired language model with the data sizes corresponding to the front-end data sizes.

6. An artificial intelligence model deployment device for GPU, characterized in that, Including: The first main module is used to determine the operator types supported by the NPU built into the GPU, convert the front-end model into an expected language model using a preset compilation framework, and locate the operators in the expected language model; the second main module is used to determine whether the operators in the expected language model belong to the operator types, remove the predetermined layers in the expected language model, merge the predetermined parameters with the bias constants of the convolutional layer to obtain a first separate convolutional layer, and combine the convolutional layer and the batch normalization layer to obtain a second separate convolutional layer. Among them, the second main module is used to traverse the operators in the expected language model layer by layer, and determine whether the operators in one layer of the expected language model belong to convolutional operators, max pooling operators, subtraction operators, addition operators, multiplication operators, or division operators; if the judgment result is yes, add the operation device of the operators in this layer as Device_NPU to the map.json file and return; if the judgment result is no, add the operation device of the operators in this layer as Device_GPU to the map.json file and return; the third main module is used to obtain the input data size and output data size of each layer in the expected language model, and determine the memory size allocated during the operation of the expected language model according to the size of the input data size and the output data size; the fourth main module is used to compile the expected language model layer by layer using a GPU compiler and an NPU compiler, and schedule the execution of the compilation results using a preset compilation framework to complete model deployment. Among them, for the operators marked as executable by Device_NPU, map them to the first binary machine code for operation in the NPU through the NPU compiler; for the operators marked as executable by Device_GPU, map them to the second binary machine code for operation in the GPU through the GPU compiler; use the TVM compilation framework to schedule the execution of the first binary machine code and the second binary machine code to complete model deployment.

7. An electronic device, characterized in that, Comprising: At least one processor, at least one memory, and a communication interface; wherein, The processor, the memory, and the communication interface communicate with each other; The memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the method according to any one of claims 1 to 5.

Images