A method and system for automatically generating inference code during model deployment.

Abstract

This invention discloses an automatic inference code generation method and system for model deployment, belonging to the field of model deployment technology. The method includes: extracting model information from intermediate files of the model to be deployed; defining variables for each operator; generating an output variable name index table and a weight variable name index table; determining the input variable names for each operator and generating an input variable name index table according to the order of the operators; generating a pad value index table, an input size index table, and an output size index table for actual inference; for each operator, extracting the corresponding API from an API library; filling the API parameters according to the output variable name index table, weight variable name index table, input size index table, output size index table, input variable name index table, and pad value index table to generate the corresponding operator's API call; and combining the API calls of each operator to obtain the required inference code. This solution provides an automatic inference code generation method for AI chips, simplifying the model deployment process.

Description

Technical Field

[0001] This solution belongs to the field of model deployment technology, and in particular relates to a method and system for automatically generating inference code in the model deployment process. Background Technology

[0002] AI model deployment includes model quantization and model inference. Currently, the mainstream AI model quantization deployment tools on the market include NVIDIA's TensorRT, Stuymin's RKNN-Toolkit, and SenseTime's OpenPPL. Their common feature is that they transform the quantized model into a custom form of their toolchain, thereby simplifying the model deployment process.

[0003] However, for AI chips that lack support from these tools, it's impossible to leverage existing quantization deployment tools to simplify development processes and reduce deployment costs. Regarding model quantization, the applicant has proposed an automated quantization deployment method and system for AI chips supporting operator-level operations [Publication No.: CN118605890B], providing an automated quantization and deployment method for AI chips like the HC3080 that are not supported by the aforementioned tools, thus improving model deployment efficiency for such chips. As for model inference, for AI chips like the HC3080 that are not supported by the aforementioned tools, the traditional approach to successfully inferring models on the AI ​​chip involves developers implementing the process layer by layer using quantization information and the operator APIs provided by the AI ​​chip. This is extremely time-consuming, especially for complex models like YOLOv7 with over two hundred operators, where the deployment process can take approximately two to three working days. Currently, there is a lack of effective means to significantly improve the efficiency of inference code implementation. Summary of the Invention

[0004] The purpose of this solution is to propose a method and system for automatically generating inference code during the model deployment process, addressing the problems existing in current technologies.

[0005] To achieve the above objectives, the following technical solutions are adopted in this proposal:

[0006] A method for automatically generating inference code during model deployment, the method comprising:

[0007] Extract model information from the intermediate files of the model to be deployed; the model information includes operator name, operator connection relationship, operator pad value and the original size of each variable of the operator;

[0008] Define the variables for each operator, including defining variable names based on operator names and memory allocation for variables based on their original dimensions. You can quickly locate the variables of an operator based on its name. The principle of memory allocation is to allocate memory for the corresponding variables by multiplying the original size by a coefficient greater than or equal to 1, such as a coefficient of 1.2.

[0009] The variables of an operator include output variables and weight variables; therefore, it includes defining the output variable name and the weight variable name. Since the input variable is the output variable of the previous operator, the input variable is not redefined. The output variable represents the input variable of the next operator.

[0010] Load the model data of the model to be deployed and generate operator weight loading code for each operator;

[0011] Generate an output variable name index table and a weight variable name index table based on the order of operators in the intermediate file and the definition of each operator variable;

[0012] The input variable name of each operator is determined based on the connection relationship of the operators, and an index table of input variable names is generated according to the order of the operators;

[0013] Based on the operator relationships and model information in the intermediate file, generate the pad value index table, input dimension index table, and output dimension index table for actual inference; two operators that are directly or indirectly connected are considered to be related.

[0014] For each operator, the corresponding API is extracted from the API library, and the parameters of the API are filled according to the output variable name index table, weight variable name index table, input size index table, output size index table, input variable name index table, and pad value index table to generate the API call for the corresponding operator.

[0015] The required inference code is obtained by combining the API calls of each operator.

[0016] In the above-described method for automatically generating inference code during model deployment, operator weight loading code is generated for each operator in the following manner:

[0017] By defining the weight data name of the corresponding operator based on the operator name, the weight variable that needs to be assigned a value can be quickly determined based on the operator name;

[0018] Generate a first bin function for the operator to read the size of the weight data;

[0019] Generate a second bin function for the operator to read weight data and assign it to the weight variable; that is, store the read weight data into the corresponding memory allocated to the weight variable.

[0020] In the above-mentioned method for automatically generating inference code during model deployment, defining variable names and weight data names based on operator names specifically includes:

[0021] For output variables, add a suffix indicating "output" to the end of the corresponding operator name to create the output variable name;

[0022] For weight variables, add a suffix indicating "weight" to the end of the corresponding operator name to serve as the weight variable name;

[0023] For weighted data, add a suffix to the end of the corresponding operator name to indicate "weighted data" as the weighted data name.

[0024] In the above-mentioned method for automatically generating inference code during model deployment, the variable names of each operator are defined in the form of pointers, so as to define the memory allocation of the variables at the same time as defining the variable names;

[0025] The generated API call also includes quantization information, fixed parameters, and operator parameters obtained from intermediate files. Quantization information is obtained during the quantization process and will not be discussed further here. Operator parameters are determined based on the specific operator and API.

[0026] In the above-mentioned method for automatically generating inference code during model deployment, the generation of inference code also includes header file inclusion and general variable definition;

[0027] After generating the inference code, this method also includes closing the file handle used to write the inference code to a C file.

[0028] In the above-mentioned method for automatically generating inference code during model deployment, generating the pad value index table, input size index table, and output size index table for actual inference based on the association relationship of operators in the intermediate file and model information specifically includes: determining the pad value of the current operator based on the model information, and generating the pad value index table according to the order of operators in the intermediate file;

[0029] The pad value is used to describe whether subsequent level operators associated with the current operator need padding and the padding size;

[0030] Based on the pad values ​​of each operator in the pad value index table and the output information of each operator in the intermediate file, i.e. the original output size, the inference output size of each operator is obtained to obtain the output size index table.

[0031] Based on the connection relationships of each operator in the intermediate file, the inference input size of each operator is obtained to obtain the input size index table.

[0032] In the above-described method for automatically generating inference code during model deployment, the pad value of the current operator is determined as follows:

[0033] Extract the operator feature list from the intermediate file. The operator feature list describes all operator features of the current model to be deployed, where each element is the operator list of the corresponding operator.

[0034] The operator list describes the current operator, including the operator name, the pad value P_INDEX of the operator in the intermediate file, and the index of the subsequent operators associated with the current operator;

[0035] Based on the operator feature list and the operator list, iterate through the operators after the current operator, and take the largest P_INDEX as the pad value of the current operator, until iterates through the convolution operator, pooling operator or the last operator of the model.

[0036] In the above-mentioned method for automatically generating inference code during model deployment, this method also includes:

[0037] Add inference error prediction code containing a code macro switch to each API call, which is used to calculate the absolute value of the difference between the inferred value and the theoretical value, and close the corresponding API call when the absolute value of the difference is greater than or equal to a preset difference value.

[0038] In the above-mentioned method for automatically generating inference code during model deployment, the intermediate file is an ONNX structure model;

[0039] This method is used to automatically generate inference code for a model to be deployed in the HC3080 chip.

[0040] An automatic inference code generation system for model deployment process is provided, which automatically generates inference code for a model to be deployed by means of the method described above.

[0041] The advantages of this solution are:

[0042] 1. This solution provides an automatic inference code generation method for AI chips such as HC3080 that are not supported by existing quantization deployment tools, simplifying the model deployment process. For example, the deployment time of a model with more than 200 operators, such as YOLOV7, can be reduced from two or three working days to a few minutes or even less than one minute.

[0043] 2. This solution proposes to automatically generate AI model inference code by obtaining key parameters of operator APIs. The key parameters required for API calls can be quickly obtained by using the operator name or index.

[0044] 3. This solution proposes a method to recursively search for the maximum pad value required by subsequent operators in intermediate files such as ONNX structures. By using this pad value and the characteristic of convolution followed by padding on specific AI chips such as HC3080, the same effect as padding followed by convolution in ONNX can be achieved, which greatly improves the model inference efficiency on AI chips such as HC3080.

[0045] 4. This solution proposes a self-checking function for inference errors while realizing automatic generation of inference code. It can calculate the inference error of each operator and provide prompts, which can ensure the accuracy of the automatically generated inference code.

[0046] 5. This solution is applicable to chips such as HC3080, enriching the ecosystem of such domestically produced chips. Attached Figure Description

[0047] Figure 1 A flowchart illustrating the method for automatically generating inference code during the model deployment process proposed in this solution.

[0048] Figure 2 This is a schematic diagram illustrating the definition method of weight variable names in an embodiment of this scheme;

[0049] Figure 3 This is a schematic diagram illustrating the method for defining variable names in this embodiment of the solution;

[0050] Figure 4 This is a diagram showing the processing sequence of padding operations for the convolution operator during ONNX model inference.

[0051] Figure 5 This is a schematic diagram of the Pad implementation of the convolution operator on the HC3080 chip in this embodiment of the solution;

[0052] Figure 6 The code example diagram shows the conv operator call and corresponding inference error calculation generated for this embodiment. Detailed Implementation

[0053] like Figure 1 As shown, this solution provides a method for automatically generating inference code during model deployment and a system for executing this method. It enables zero-click generation of inference code, such as C code, for AI models on AI chips. In the model deployment process, it is typically necessary to convert the model to be deployed into an intermediate format. Currently, the widely used intermediate representation format is ONNX (Open Neural Network Exchange), which allows models to be shared between different deep learning frameworks. This embodiment uses ONNX as an example intermediate format; however, other feasible intermediate formats can be used instead, and this is not a limitation.

[0054] This embodiment uses the typical HC3080 chip as an example to describe in detail the automatic inference code generation process. The principle of automatic code generation is simple: create a file handle and write the planned code string into a file with the ".c" extension. Therefore, obtaining the code string is the most critical issue. This solution divides the C code for model inference into five main parts: header file inclusion and general variable definition, operator input / output and weight variable definition, loading the input file and weight file, inference code generation, and file handle release.

[0055] Header file inclusion and general variable definition

[0056] The purpose of this embodiment is to generate C code for an AI model that can be used for inference on the HC3080. Since the beginning of C language is to include relevant header files and define common variables, this embodiment first completes the automatic generation of this part of the code.

[0057] #include "XD_CMODEL.h"

[0058] #include ".. / lib / vsip.h"

[0059] #define DEBUG_DIFF

[0060] long long t_time = 0;

[0061] long long t0=0;

[0062] long long t1=0;

[0063] u32 RespType;

[0064] u32 size;

[0065] The above is a code example of header file inclusion and general variable definition in this embodiment of the solution. The first two lines are header file inclusion, the third line defines a macro switch, and the last four lines define general global variables, such as those used for timing or statistical quantification of data size. This code is uniform and fixed for each AI model.

[0066] Operator input / output and weight variable definition

[0067] In this embodiment, the definition of operator-related variables follows the following rules:

[0068] The operator names are parsed based on the ONNX structure, and the variable names of each operator in the inference code are defined according to the operator names. Below is an example of operator variable definition code given in this embodiment:

[0069] u64 block1_conv1_Conv_out=0x1080000000+134912;

[0070] u64 block1_conv1_Conv _wb=0x1080000000+245056;

[0071] u64 block1_conv2_Conv _out=0x1080000000+258144;

[0072] u64 block1_conv2_Conv _wb=0x1080000000+387616;

[0073] u64 block1_Add _out=0x1080000000+400448;

[0074] For a model to be deployed, an operator in its ONNX structure is named `block1_conv1_Conv`. After obtaining this operator name, the weight variable name for this operator is defined as `block1_conv1_Conv` with the suffix "_wb", and the output variable name is defined as `block1_conv1_Conv` with the suffix "_out". Since the operator input is always the output of another operator, no further operator input variables are defined here. As shown in the operator variable code example above, this solution defines operator-related variables in pointer form to achieve reasonable memory allocation. "0x1080000000" is the starting memory address, and the offset numbers added after it can be calculated based on the output matrix or weight matrix. The difference between the offset number of the next variable and the offset number of the current variable represents the memory size allocated to the current variable. The ONNX structure contains the dimensions of each operator output and weight, so these offsets can be calculated and this part of the code can be generated relatively easily. The dimensions here are the original dimensions obtained from ONNX. If an operator needs to pad the output, the output variable dimensions during inference are re-determined based on the pad value.

[0075] Load input file and weight file

[0076] After defining the variables, load the model data for the model to be deployed, which mainly includes weight data and model input data. Below is a sample code for loading operator weights in this implementation example:

[0077] size = get_bin_size(" / ffx0 / models / bins / block1_conv1_Conv_wb.bin");

[0078] read_bin(" / ffx0 / models / bins / block1_ conv1_conv_wb.bin",block1_ conv1_Conv_wb, size);

[0079] size = get_bin_size(" / ffx0 / models / bins / block1_ conv2_conv_wb.bin");

[0080] read_bin(" / ffx0 / models / bins / block1_ conv2_conv_wb.bin",block1_ conv2_Conv_wb, size);

[0081] Assuming the operator name parsed from the ONNX structure is `block1_conv1_Conv` as shown in the operator weight code example above, then the weight data name is defined as `block1_conv1_Conv` with the suffix "_wb.bin", and the weight variable name is `block1_conv1_Conv` with the suffix "_wb". The weight data name and weight variable name are obtained based on the operator name. Simultaneously, a first bin function—`get_bin_size`—is generated to simply read the size of the bin file, and a second bin function—`read_bin`—is generated to read data from the bin file and assign it to the weight variables. In this way, operator weight loading code is automatically generated based on the operator name.

[0082] Inference code generation

[0083] This embodiment generates inference code based on the ONNX structure of the AI ​​model. Common operators included in the ONNX model include activation functions such as conv, gemm, and relu, pooling functions such as maxpool, and operators such as unsqueeze, flatten, transpose, resize, add, and concat. The HC3080 chip provides APIs for some operators, such as activation functions like conv, gemm, and relu, and pooling functions like maxpool. For operators for which the HC3080 chip provides APIs, the chip's APIs are used to generate the operator's inference code. For operators for which the HC3080 chip does not provide APIs, this solution provides a set of APIs implemented in C or other feasible languages ​​for generating the corresponding operator's inference code. The API library used to generate the inference code includes the APIs provided by the AI ​​chip and the set of APIs implemented in C or other feasible languages.

[0084] The process of generating inference code involves calling all operators sequentially according to the order of operations in the ONNX structure. The output of the current operator is passed to the next operator it is associated with, and the output of the last operator becomes the final inference result. Every operator call inevitably involves input and output variables; weighted operators also involve weight variables. These input and output weights are stored in corresponding variables, specifically in the memory space allocated in the "Operator Input / Output and Weight Variable Definitions" section above. The operator API calls relevant variables based on operator relationships and stores the operator prediction results in the output variables.

[0085] This scheme has defined the associations of each variable based on the operator name, that is, as follows: Figure 2 As shown, for the operator `conv`, the weight variable name is defined as the operator name `conv` followed by the suffix "_wb". The weight variable name is unique and distinct for each operator; weights are defined according to the above rules if they exist, otherwise no weight variable is defined. Similarly, the output variable name is defined as the operator name `conv` followed by the suffix "_out". The output variable name is also unique and distinct for each operator. Furthermore, this scheme adds the weight variable names to the weight variable name index table—the `global_list_wb` list—in order of operator execution. For operators without weights, a 0 is filled in the corresponding position in the list to distinguish them. Similarly, the output variable names are added to the output variable name index table—the `global_list_out` list—in order of operator execution.

[0086] The input variable name is determined by the output variable name of the associated operator, such as... Figure 3 As shown, different operators may have the same input variables, and an operator may have multiple input variables. The input variable name for each operator can be determined based on the connection relationships between operators in the ONNX structure. Similarly, the input variable names are added to the input variable name index table—global_list_in. In this way, the input and output weight variable names of the operator can be quickly found based on the operator name or operator index.

[0087] The definition of a general operator API is usually in the following format:

[0088] operator_api(input_var,weight_var,output_var,input_shape,output_shape,pad)

[0089] There may be more parameters, but the following six should be considered first: input_var, weight_var, and output_var are the input variable, weight variable, and output variable, respectively. input_shape is the size of the input variable during inference, determining the scale of the data involved in the calculation. output_shape is the size of the output variable during inference, which must be used when calculating the inference error of this operator, as mentioned later. Output_shape is also the input size of the associated operator during inference. Pad values ​​will change the size of the input or output, especially since the padding method on the HC3080 differs from the conventional method, making this particularly important.

[0090] Pad generally refers to the number of pads in a convolutional or pooling layer, such as Figure 4When inferring a convolution operator with a pad of 1 in ONNX models, the calculation order is to first pad the input to 1*47*34*34, and then feed the padded input into the convolution-related API for inference. If this order is followed during inference on the HC3080, the inference time will be significant because padding operations require C language implementation. While the HC3080's convolution operators do not support padding before computation, they allow padding one or more loops of 0 in the output. When this convolution operator is followed by another convolution operator or pooling operator (activation functions like ReLU can be skipped without changing the input / output dimensions), the padding in the output prepares for the padding in the next operator, thus avoiding the padding effect seen in C language. Figure 5 On the HC3080, the pad of the current operator is implemented using the output pad of the previous conv operator, that is... Figure 5 The output of the first conv is already 1*47*34*34.

[0091] This solution leverages the characteristic of the conv operator on the HC3080 that it performs operations before padding. When calling the conv operator, a pad parameter is determined first. This parameter does not describe whether the current operator needs padding, but rather whether the next level operator or several levels below it needs padding, and how much padding is required. Therefore, the pad value needs to be determined by traversing the operators after the current conv operator according to the onnx structure. The pseudocode example for determining the pad value in this implementation is as follows:

[0092] Define the function get_pad(inlist, oplist, max_pad);

[0093] If the first element of inlist contains "conv"

[0094] Returns the padding index (p_INDEX) of the inlist.

[0095] If the first element of inlist contains "pool"

[0096] Return 0

[0097] If the next data node (DATA_NEXT) of inlist is empty

[0098] Returns the padding index (p_INDEX) of the inlist.

[0099] Iterate through all the next data nodes (DATA_NEXT) in the inlist.

[0100] Calculate index j as the current node - 1 (index starts from 0).

[0101] Recursively call the get_pad function to obtain pad_comp

[0102] If pad_comp is greater than max_pad

[0103] Update max_pad to pad_comp

[0104] Return max_pad

[0105] The operator list `inlist` is a list describing the current operator, containing the operator name, the operator's pad value in `onnx` (`P_INDEX`), and the index of the next operator associated with the current operator (`DATA_NEXT`). The operator feature list `oplist` is a list describing all operator features, where each element is an `inlist` list of some operator. `max_pad` represents the maximum pad value currently encountered. This recursive function terminates under three conditions: encountering a `conv` operator, encountering the last operator in the model whose `DATA_NEXT` is empty, and encountering a `pool` operator. Encountering a `pool` operator returns a pad value of 0 because the `maxpool` operator requires padding with the minimum input value, not 0, but on the HC3080, `conv` operators can only be filled with 0; therefore, no padding is performed when encountering a `pool` operator. After calculating the pad value, it is added to the pad value index table—the `global_list_real_pad` list.

[0106] The pad value for each operator in the model is determined in the above manner. For general operators, the pad is used to calculate the size of the output variable during inference. For convolution and pooling operators, the API parameters include this pad value.

[0107] Based on the pad values ​​in the global_list_real_pad list and the operator output information in the ONNX model, the output_shape of each operator is calculated:

[0108] H new =H old +2*pad

[0109] W new =W old +2*pad

[0110] H new W new These represent the height and width of the output variable during operator inference, respectively;

[0111] H old W old These represent the height and width of the operator output variable in ONNX, respectively.

[0112] Similarly, add them sequentially to the output size index table—global_list_real_output. Based on the connection relationship of operators in the ONNX structure, similar to obtaining the input variable name, obtain the input_shape of each operator, and similarly add them sequentially to the input size index table—global_list_real_input.

[0113] Thus, this scheme designs six global lists: global_list_in, global_list_wb, global_list_out, global_list_real_pad, global_list_real_input, and global_list_real_output. These lists store the input variable name, weight variable name, output variable name, operator pad value, input size, and output size required for operator calls, in the order of the operators. Based on these global lists, the parameter information required for operator calls, i.e., the key API parameters, can be obtained.

[0114] Because operator inference is performed sequentially, errors in the calculation of earlier operators will affect the inference of subsequent operators. Therefore, this solution adds a function to the deployment code to calculate the inference error of each operator. Based on the inference error of each operator, it can quickly confirm whether the inference is correct or which operator's inference is wrong. Figure 6 The first four lines are the standard conv operator calls on the HC3080, and the last four lines are the code for calculating the inference error. compare_rlt_offset_diff is the corresponding function. The first parameter is the amount of data involved in the statistics, the second parameter indicates how many bytes each number is described in, the third parameter indicates the address of the predicted value, the fourth parameter indicates how many numbers to offset each time, and the fifth parameter indicates the theoretical output value calculated using the ONNX model.

[0115] The following is an example of the inference error calculation results in this embodiment of the solution:

[0116] compare_rlt start.

[0117] points 1600 total diff 961 max value of diff 2

[0118] cos similiraty : 1.000000

[0119] Finally, the compare_rlt_offset_diff function will output the cosine similarity between the predicted and theoretical values ​​(cos_similiraty), the maximum absolute difference between the predicted and theoretical values ​​across all points being compared (max_value_of_diff), and the sum of the absolute differences between the predicted and theoretical values ​​across all points being compared (total_diff). Generally, if max_value_of_diff is less than 100 and cos_similiraty is greater than 0.999, the inference error is considered acceptable.

[0120] Thus, the above method has enabled the generation of six global lists required for API calls based on the ONNX model structure. Combined with the quantization information of the operators, inference code can be automatically and quickly generated (generally 3 to 9 truncation values; how the quantization information is generated is not discussed here). For example... Figure 6 Taking the convolution API on the HC3080 as an example, the information obtained above already includes the parameter information contained in the nine boxes. The first is the operator input / output size, obtained from the `global_list_real_input` and `global_list_real_output` lists. Data in the same order is retrieved from these two lists according to the current convolution operator order. The second is the number of convolution input / output channels, obtainable from the ONNX file. The third is the pad parameter, obtained from the `global_list_real_pad` list; similarly, data in the same order is retrieved from this list according to the current convolution operator order. The fourth, fifth, and sixth are the quantization parameters, obtained from the quantization information. The seventh parameter is the input variable name, the eighth is the weight variable name, and the ninth is the output variable name. These three parameters are obtained from the three lists: `global_list_in`, `global_list_wb`, and `global_list_out`, respectively. Variable names can be retrieved from these lists based on the operator name. During inference, the weights of the corresponding operators are extracted by the weight loading code and stored in the corresponding memory allocation space for inference. Data, including input data and weight data, is extracted from the memory allocation space of the corresponding variable based on the variable name. Simultaneously, the inference result (i.e., output data) of the convolution operator is stored in the memory allocation space of the corresponding variable. The next connection operator using this output as input extracts data from this memory allocation space. Most of the remaining parameters are fixed and are called fixed parameters. A few parameters, such as the convolution kernel size, can be directly obtained from the ONNX structure and are called operator parameters. Operator parameters are determined based on the specific operator and API. Since the parameter positions are fixed, API call code can be automatically and efficiently generated. Combining the operator inference error calculation logic described above, this part of the code can also be automatically generated. The final generated model inference code includes API calls for each operator and operator inference error calculation code.

[0121] Release file handle

[0122] The generation of C code actually involves writing the above code into a C file using a file handle. Therefore, the file handle must be closed after writing the code. Furthermore, if this automatic deployment code generation function is provided to the user in a GUI format, the program will inevitably generate C code for different models repeatedly after startup. Therefore, after each generation of C code, the contents of the above six global lists must be cleared to prevent the operator information of the previous model from affecting the deployment code generation of the current model. An example of file handle release and global list clearing in this embodiment is shown below:

[0123] c_f.close()

[0124] global_list_in.clear()

[0125] global_list_out.clear()

[0126] global_list_wb.clear()

[0127] global_list_real_pad.clear()

[0128] global_list_real_output.clear()

[0129] global_list_real_input.clear()

[0130] The inference code implemented above, during the inference process, when it executes... Figure 6 When performing the convolution operator shown, the weight variable name, input variable name, and output variable name are obtained through the convolution API. Then, based on the input variable name and weight variable name, the input data is extracted from the storage space of the corresponding input variable, the weight data is extracted from the storage space of the corresponding weight variable, and the output data is stored in the storage space with the corresponding variable name. After all operators are executed in sequence, the inference ends.

[0131] The specific embodiments described herein are merely illustrative examples of the spirit of this solution. Those skilled in the art to which this solution pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of this solution or exceeding the scope defined by the appended claims.

Claims

1. A method for automatically generating inference code during model deployment, characterized in that, For AI chips that have the characteristic of computation before filling, this method includes: Extract model information from the intermediate file of the model to be deployed; the model information includes operator name, operator connection relationship, operator pad value and the original size of each variable of the operator; the intermediate file is an ONNX structure model; Define the variables for each operator, including defining variable names based on operator names and memory allocation for variables based on their original dimensions; The variables of an operator include output variables and weight variables; Load the model data of the model to be deployed and generate operator weight loading code for each operator; Generate an output variable name index table and a weight variable name index table based on the order of operators in the intermediate file and the definition of each operator variable; The input variable name of each operator is determined based on the connection relationship of the operators, and an index table of input variable names is generated according to the order of the operators; Based on the operator relationships and model information in the intermediate file, generate the pad value index table, input dimension index table, and output dimension index table for actual inference: The pad value of the current operator is determined based on the model information, and a pad value index table is generated according to the order of the operators in the intermediate files. The pad value is used to describe whether subsequent level operators associated with the current operator need padding and the padding size; The output size index table is obtained by acquiring the inference output size of each operator based on the pad value of each operator in the pad value index table and the original output size of each operator in the intermediate file. Based on the connection relationships of each operator in the intermediate file, the inference input size of each operator is obtained to obtain the input size index table; The pad value of the current operator is determined as follows: Extract the operator feature list from the intermediate file. The operator feature list describes all operator features of the current model to be deployed, where each element is the operator list of the corresponding operator. The operator list describes the current operator, including the operator name, the pad value P_INDEX of the operator in the intermediate file, and the index of the subsequent operators associated with the current operator; Based on the operator feature list and the operator list, iterate through the operators after the current operator, and take the largest P_INDEX as the pad value of the current operator, until iterates through the convolution operator, pooling operator or the last operator of the model; For each operator, the corresponding API is extracted from the API library, and the parameters of the API are filled according to the output variable name index table, weight variable name index table, input size index table, output size index table, input variable name index table, and pad value index table to generate the API call for the corresponding operator. The required inference code is obtained by combining the API calls of each operator.

2. The method for automatically generating inference code during model deployment according to claim 1, characterized in that, The operator weight loading code is generated for each operator in the following manner: Define the weight data name of the corresponding operator based on the operator name; Generate a first bin function for the operator to read the size of the weight data; Generate a second bin function for the operator to read the weight data and assign it to the weight variable.

3. The method for automatically generating inference code during model deployment according to claim 2, characterized in that, The variable names and weight data names defined according to the operator names specifically include: For output variables, add a suffix indicating "output" to the end of the corresponding operator name to create the output variable name; For weight variables, add a suffix indicating "weight" to the end of the corresponding operator name to serve as the weight variable name; For weighted data, add a suffix to the end of the corresponding operator name to indicate "weighted data" as the weighted data name.

4. The method for automatically generating inference code during model deployment according to claim 1, characterized in that, Define the variable names of each operator in the form of pointers, so as to define the memory allocation of the variables at the same time as defining the variable names; The generated API calls also include quantization information, fixed parameters, and operator parameters obtained from intermediate files.

5. The method for automatically generating inference code during model deployment according to claim 1, characterized in that, The generation of inference code also includes header file inclusion and general variable definition; After generating the inference code, this method also includes closing the file handle used to write the inference code to a C file.

6. The method for automatically generating inference code during model deployment according to claim 1, characterized in that, This method also includes: Add inference error prediction code containing a code macro switch to each API call, which is used to calculate the absolute value of the difference between the inferred value and the theoretical value, and close the corresponding API call when the absolute value of the difference is greater than or equal to a preset difference value.

7. The method for automatically generating inference code during model deployment according to claim 1, characterized in that, This method is used to automatically generate inference code for a model to be deployed in the HC3080 chip.

8. An automatic inference code generation system for a model deployment process, characterized in that it is used to automatically generate inference code for a model to be deployed by means of the method described in any one of claims 1-7.

Images