Electronic device and method for optimization of an artificial intelligence inference model

Abstract

The present disclosure relates to an electronic device and method for optimization of an artificial intelligence (AI) model. An electronic device for optimization of an AI inference model, the electronic device comprising: a processing circuit, the processing circuit being configured to: extract an AI inference model to be optimized; acquire an optimization requirement for the AI inference model, the optimization requirement being related to one or more of the following configurations for an end side device in which the AI inference model is to be deployed: an operator range supported by the end side device, a computing capability of the end side device, and an output bandwidth limit of the end side device; and optimize the network structure of the AI inference model on the basis of the optimization requirement to obtain an optimized AI inference model.

Description

Electronic device and method for optimizing artificial intelligence inference models

[0001] Priority Declaration

[0002] This application claims priority to the Chinese patent application filed on September 26, 2023, with application number 202311253414.0 and entitled “Electronic device and method for optimization of artificial intelligence reasoning models,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computers, and more specifically, to the optimization of artificial intelligence reasoning models. Background Art

[0004] Inference of an artificial intelligence (AI) model refers to deploying an AI model that has been pre-trained on the server side, for example, to an end-side device, where the AI ​​inference model infers the input data to obtain the corresponding output data.

[0005] As the network structure of AI models becomes more complex, the AI ​​inference models deployed on end-side devices may also have complex network structures. However, the hardware configuration of end-side devices (such as computing power, storage capacity, input and output bandwidth, etc.) is limited, which may lead to restrictions on the deployment of AI inference models on end-side devices. Therefore, there is a need to optimize AI inference models.

[0006] Summary of the Invention

[0007] A brief overview of the present disclosure is provided below to provide a basic understanding of some aspects of the present disclosure. However, it should be understood that this overview is not an exhaustive overview of the present disclosure. It is not intended to identify key or important parts of the present disclosure, nor is it intended to limit the scope of the present disclosure. Its purpose is simply to present certain concepts of the present disclosure in a simplified form as a prelude to the more detailed description that will be given later.

[0008] According to one aspect of the present disclosure, an electronic device for optimizing an artificial intelligence (AI) reasoning model is provided, the electronic device comprising: a processing circuit, the processing circuit being configured to: extract an AI reasoning model to be optimized; obtain optimization requirements for the AI ​​reasoning model, the optimization requirements being related to one or more of the following configurations of an end-side device where the AI ​​reasoning model will be deployed: a range of operators supported by the end-side device, the computing power of the end-side device, and an output bandwidth limit of the end-side device; and optimizing the network structure of the AI ​​reasoning model according to the optimization requirements to obtain an optimized AI reasoning model.

[0009] According to another aspect of the present disclosure, a method for optimizing an artificial intelligence (AI) reasoning model is provided, including: extracting an AI reasoning model to be optimized; obtaining optimization requirements for the AI ​​reasoning model, wherein the optimization requirements are related to one or more of the following configurations of an end-side device where the AI ​​reasoning model will be deployed: a range of operators supported by the end-side device, the computing power of the end-side device, and an output bandwidth limit of the end-side device; and optimizing the network structure of the AI ​​reasoning model according to the optimization requirements to obtain an optimized AI reasoning model.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided, comprising executable instructions. When the executable instructions are executed by an information processing device, the information processing device is caused to perform the method according to the present disclosure.

[0011] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the method according to the present disclosure.

[0012] According to one or more embodiments of the present disclosure, the AI ​​inference model can be adaptively optimized based on the hardware configuration of the end-side device where the AI ​​inference model will be deployed, without modifying the hardware configuration of the end-side device, so that the optimized AI inference model can adapt to the performance of the end-side device and achieve good inference results. In addition, the optimization of the AI ​​inference model of the present application does not require the use of user data of the end-side device to retrain the AI ​​inference model, and can be applied to end-side devices that are sensitive to user data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0014] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0015] FIG1 is a block diagram illustrating an exemplary configuration of an electronic device for optimizing an AI reasoning model according to an embodiment of the present disclosure;

[0016] FIG2 is an exemplary flowchart illustrating a method for optimizing an AI reasoning model according to an embodiment of the present disclosure;

[0017] FIG3 is an exemplary flowchart illustrating a method for optimizing an AI reasoning model according to another embodiment of the present disclosure;

[0018] FIG4 is an exemplary flowchart illustrating a method for optimizing an AI reasoning model according to another embodiment of the present disclosure;

[0019] FIG5 illustrates an exemplary configuration of a computing device in which embodiments according to the present invention may be implemented. DETAILED DESCRIPTION

[0020] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0021] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0022] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0023] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0026] For example, AI models can be pre-trained on a server with abundant computing and storage resources. When a pre-trained AI model is deployed on an edge device as an AI inference model, the hardware configuration of the edge device (e.g., computing power, storage capacity, input-output (IO) bandwidth, etc.) may be limited. Therefore, the deployment of the AI ​​inference model on the edge device may be subject to one or more of the following limitations.

[0027] I / O bandwidth limitation: The bus bandwidth of the edge device to be deployed may be much smaller than the bandwidth of the server used for AI model training. As a result, the I / O bandwidth (for example, output bandwidth) may be limited when the AI ​​inference model is applied on the edge device, affecting the performance of the AI ​​inference model and even preventing the AI ​​inference model from running normally on the edge device.

[0028] Operator type restrictions: The terminal device to be deployed can only support a limited number of operator types and may not be able to support all operators involved in the AI ​​inference model. As a result, the AI ​​inference model cannot run normally after it is deployed on the terminal device. Operator types include, for example, convolution, pooling, activation units, addition and subtraction, multiplication and division, absolute value, and non-maximum suppression (NMS). The server side used for AI model training may support a wider range of operators, while the terminal device may support fewer types of operators due to the limitations of the hardware platform capabilities, and different terminal devices may support different types of operators.

[0029] Computing power: Due to the limited computing resources and memory of the end-device, as well as possible constraints related to computing power such as fixed-point limitations, the computing power of the end-device is lower than that of the server used for AI model training. Therefore, when using the AI ​​inference model for inference on the end-device, it may not be possible to achieve the same accuracy as the server.

[0030] Customized platform dependency: Some AI models rely on or work with specific hardware (such as customized hardware platforms). When deployed on non-customized edge devices, they may not achieve ideal performance.

[0031] Limited user data: When deploying AI inference models to edge devices, it is often necessary to retrain the AI ​​inference models using the device's user data and modify the model's structure or parameters to meet performance requirements. However, in some cases, user data is private and sensitive (e.g., patient information in hospital systems, facial information in public security systems, and text / data scans in financial companies) and cannot be used for retraining, making it impossible to adaptively optimize the AI ​​inference model.

[0032] If you are subject to one or more of the above restrictions, you may need to change your hardware configuration (such as replacing the hardware processor or changing the hardware platform) or agree to use user private data, depending on the specific restrictions, to mitigate the potential adverse effects of the above restrictions. However, changing hardware configuration is costly, and using user private data may raise security concerns.

[0033] In response to one or more of the above problems, the present disclosure proposes an electronic device and method for optimizing an AI reasoning model. The device can adaptively optimize the AI ​​reasoning model based on the hardware configuration of the end-side device where the AI ​​reasoning model will be deployed, without modifying the hardware configuration of the end-side device, so that the optimized AI reasoning model can adapt to the performance of the end-side device and achieve good reasoning results. In addition, the optimization of the AI ​​reasoning model of the present application does not require the use of user data of the end-side device to retrain the AI ​​reasoning model, and can be applied to end-side devices that are sensitive to user data.

[0034] Next, referring to FIG. 1 and FIG. 2 , an electronic device and method for optimizing an AI reasoning model according to an embodiment of the present disclosure will be described in detail.

[0035] 1 shows an exemplary configuration block diagram of an electronic device 1000 according to an embodiment of the present disclosure. The electronic device 1000 can be used, for example, to optimize an AI reasoning model.

[0036] In some embodiments, the electronic device 1000 may include a processing circuit 1010. The processing circuit 1010 of the electronic device 1000 provides various functions of the electronic device 1000. In some embodiments, the processing circuit 1010 of the electronic device 1000 may be configured to perform a method for the electronic device 1000.

[0037] Processing circuitry 1010 may refer to various implementations of digital circuitry, analog circuitry, or mixed-signal (a combination of analog and digital) circuitry that performs functions in a computing system. Processing circuitry may include, for example, circuits such as integrated circuits (ICs), application-specific integrated circuits (ASICs), portions or circuits of a separate processor core, an entire processor core, a separate processor, a programmable hardware device such as a field programmable gate array (FPGA), and / or a system including multiple processors.

[0038] In some embodiments, the processing circuit 1010 may include an AI model extraction unit 1020, an optimization requirement acquisition unit 1030 and an AI model optimization unit 1040, configured to execute corresponding steps in the method 2000 shown in Figure 2 described below.

[0039] In some embodiments, the electronic device 1000 may further include a memory (not shown). The memory of the electronic device 1000 may store information generated by the processing circuit 1010 as well as programs and data used for the operation of the electronic device 1010. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory.

[0040] In addition, the electronic device 1000 may be implemented at a chip level, or may be implemented at a device level by including other external components. In some embodiments, the electronic device 1000 may be implemented as a complete device, and may further include multiple antennas.

[0041] It should be understood that the above-mentioned units are merely logical modules divided according to the specific functions they implement, and are not intended to limit the specific implementation methods. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.).

[0042] FIG2 shows an exemplary flowchart of a method 2000 for optimizing an AI reasoning model according to an embodiment of the present disclosure. The method 2000 may be implemented by the electronic device 1000 shown in FIG1 , for example.

[0043] As shown in FIG. 2 , in S2010 , the AI ​​model extraction unit 1020 extracts the AI ​​reasoning model to be optimized.

[0044] In some embodiments, the AI ​​reasoning model to be optimized may be, for example, an AI model that has been pre-trained on the server side, the cloud, etc. and stored in a corresponding database. The corresponding AI reasoning model to be optimized is extracted by accessing the corresponding database. The AI ​​reasoning model may be any model based on an artificial intelligence algorithm, for example, a model based on a neural network, etc. In addition, the type of the AI ​​reasoning model may be, for example, a model for scenarios such as image recognition, feature extraction, text / speech recognition, or a model applied to any other scenario. In addition, the AI ​​reasoning model involved in the present disclosure may be stored in, for example, tflite format or tensorrt format, or other AI model formats may be used. Depending on the format adopted, corresponding tools are used to extract and parse the AI ​​reasoning model.

[0045] In S2020, the optimization requirement acquisition unit 1030 acquires the optimization requirements for the AI ​​reasoning model extracted in S2010. In some embodiments, the optimization requirements may be related to the hardware configuration of the end-side device where the AI ​​reasoning model will be deployed. The hardware configuration is, for example, one or more of the following configurations: the range of operators supported by the end-side device, the computing power of the end-side device, and the output bandwidth limit of the end-side device. In the present disclosure, the end-side device is, for example, a mobile terminal, a personal computer, a tablet, a smart watch, and other terminal devices. In addition, the end-side device may also be other devices such as an edge computing device, as long as it is a device that needs to deploy a pre-trained AI model for reasoning.

[0046] In some embodiments, the optimization requirements can be obtained from the end-side device where the AI ​​reasoning model will be deployed and stored in the corresponding configuration file. In addition, the optimization requirements can be specified by the user of the end-side device to which the AI ​​reasoning model will be applied. In some embodiments, the configuration file can, for example, record one or more of the operator type of the end-side device, the memory limit that can be used for the AI ​​reasoning model, the IO bandwidth limit, the throughput limit, the precision limit, etc. In addition, other optimization requirements related to the configuration of the end-side device can also be specified as needed. The above-mentioned restrictions recorded in the configuration file can be used to determine the range of operators supported by the end-side device, the computing power of the end-side device (for example, determined according to memory limits, precision limits, etc.), and the output bandwidth limit of the end-side device.

[0047] In S2030, the AI ​​model optimization unit 1040 optimizes the network structure of the AI ​​reasoning model according to the optimization requirements obtained in S2020 to obtain an optimized AI reasoning model. As a result, the optimized AI reasoning model can be deployed on the end-side device.

[0048] According to the method 2000 disclosed herein, the AI ​​inference model can be adaptively optimized based on the hardware configuration of the end-side device where the AI ​​inference model will be deployed, without modifying the hardware configuration of the end-side device, so that the optimized AI inference model can adapt to the performance of the end-side device and achieve good inference results. In addition, the optimization of the AI ​​inference model of the present application does not require the use of user data of the end-side device to retrain the AI ​​inference model, and can achieve offline optimization of the AI ​​inference model, which is suitable for end-side devices that are sensitive to user data.

[0049] Next, referring to Figure 3, an exemplary flowchart of a method 3000 for optimizing an AI reasoning model according to another embodiment of the present disclosure is described. The method 3000 can be implemented, for example, by the electronic device 1000 shown in Figure 1, wherein each step can be implemented, for example, by configuring the processing circuit 1010 accordingly.

[0050] As shown in FIG3 , at S3010, the AI ​​reasoning model is parsed and optimization requirements for the AI ​​reasoning model are obtained. The AI ​​reasoning model may be, for example, the AI ​​reasoning model to be optimized extracted at S2010 of method 2000. In addition, the optimization requirements may be, for example, obtained at S2020 of method 2000.

[0051] In some embodiments, the AI ​​reasoning model to be optimized can be parsed to determine the network structure of the AI ​​reasoning model. In addition, a configuration file recording the optimization requirements can also be parsed to obtain the optimization requirements.

[0052] In some embodiments, parsing tools corresponding to the format of the AI ​​model can be used for parsing. For example, when the AI ​​reasoning model is in tflite format, the flatbuffer toolkit can be used to parse the AI ​​reasoning model file into a json-similar format to read the network structure of the AI ​​reasoning model. In some embodiments, the network structure of the AI ​​reasoning model can be represented by a computational graph, including nodes, connection relationships between nodes, and weights.

[0053] In some embodiments, in S3020, the AI ​​inference model may be structurally re-parameterized based on optimization requirements to reduce the number of branches in the network structure of the AI ​​inference model, thereby matching the optimized AI inference model with the computing power of the end-side device. Structural re-parameterization refers to rearranging the parameters of the AI ​​inference model to change the network structure of the AI ​​inference model without changing the content of the parameters.

[0054] In some embodiments, structural reparameterization can be used to convert the AI ​​reasoning model of the multi-branch structure into a single-branch network structure, thereby more efficiently utilizing the computing resources of the end-side device, saving memory space, and improving operational efficiency. In addition, the present disclosure is not limited to converting the multi-branch network structure into a single-branch network structure. The number of branches can also be reduced to more than one branch based on the computing power of the end-side device (for example, memory limitations, precision loss limitations, fixed-point limitations, etc. that can be used for the AI ​​reasoning model). Accordingly, the weights of the multi-branch structure are also rearranged to the corresponding nodes of the single-branch structure without changing the specific value of the weight.

[0055] In some embodiments, the size of the convolution kernel in the network structure of the AI ​​reasoning model can be further reduced by structurally reparameterizing the AI ​​reasoning model, thereby further improving the throughput (the number of frames per second for reasoning) and processing efficiency. In some embodiments, the size of the convolution kernel can be appropriately designed according to the hardware configuration of the end-side device (such as the data alignment method in the hardware platform). For example, the size of the convolution kernel can be determined based on the data alignment method in the hardware platform of the end-side device to reduce operations such as data shifting, thereby improving processing efficiency. In addition, the size of the convolution kernel can also be determined based on the parallelism of the hardware platform of the end-side device (the number of channels that can simultaneously process data). The data alignment method and the parallelism of the hardware platform described above can be recorded in the configuration file as optimization requirements, so as to refer to these optimization requirements for structural reparameterization, thereby adjusting the network structure of the AI ​​reasoning model to match the hardware configuration (computing power) of the end-side device.

[0056] In some embodiments, the AI ​​inference model can be reparameterized to optimize its network structure into a single-branch structure with a 3×3 convolution and activation function structure (3×3Conv+ReLu). Since 3×3Conv+ReLu is the smallest convolution kernel operation unit in the AI ​​inference model, it can further simplify the structure of the AI ​​inference model, reduce operating memory costs, and accelerate the parallel processing efficiency of inference.

[0057] In some embodiments, the AI ​​reasoning model to be optimized is parsed in S3010, and operators required to run the AI ​​reasoning model to be optimized may be determined. Next, in some embodiments, operators of the AI ​​reasoning model to be optimized may be replaced according to optimization requirements in S3030.

[0058] In some cases, the types of operators supported by the end-side device to be deployed may be limited, and it may not support all operators involved in the AI ​​inference model to be optimized. For example, through analysis, it is determined that the operators involved in the AI ​​inference model to be optimized include convolution, pooling, activation unit, addition and subtraction, multiplication and division, absolute value, and NMS, while the operators supported by the end-side device determined based on the optimization requirements only include convolution, pooling, activation unit, addition and subtraction, and multiplication and division. In other words, the end-side device does not support absolute value and NMS.

[0059] In such cases, operators within the range of operators supported by the end-side device can be used to replace the operators required to run the AI ​​inference model to be optimized. For example, for an absolute value operator not supported by the end-side device, corresponding calculation rules can be designed using the addition and subtraction operators supported by the end-side device to implement absolute value operations, thereby replacing the unsupported absolute value operator.

[0060] As another example, the NMS operator can be used to eliminate the influence of adjacent points in the inference results on the judgment of extreme points in non-continuous areas, and retain the extreme points in the valid area. However, this type of NMS operator (such as TFLite_Detection_PostProcess) may not be supported by all AI hardware platforms, especially for some device-side low-power and low-computing platforms. Therefore, in this disclosure, for this situation, corresponding calculation rules can be designed using operators supported by the end-side device (such as MaxPool2D, Sub, Add, ReLu, MuL, etc.) to implement NMS operations, thereby replacing unsupported NMS operators.

[0061] More specifically, the following operator replacement method may be used to eliminate adjacent interference points.

[0062] 1) For the pixels in the original heatmap, subtract the corresponding values ​​after the maximum pooling (MaxPool2D) so that the original local extreme point positions are 0 and the non-extreme point positions are small negative values.

[0063] 2) Add the result obtained in the previous step to the positive minimum value, so that the original local extreme point position is the positive minimum value and the non-extreme point position is the negative minimum value.

[0064] 3) Perform ReLu activation operation on the result obtained in the previous step, so that the original local extreme point position is a positive minimum value and the non-extreme point position is 0.

[0065] 4) Multiply the result obtained in the previous step by the (MUL) magnification factor, so that the positive minimum value of the original local extreme point position becomes 1, and the non-extreme point position remains 0.

[0066] 5) Multiply the result obtained in the previous step by (MUL) the value of the corresponding pixel in the original heat map. The extreme point position is the corresponding pixel value of the original heat map, and the non-extreme point position is still 0.

[0067] In addition, for the case where the maximum value in the heat map is less than 0, the following operator replacement method can be used.

[0068] 6) Invert the result obtained in step 3) above (MUL).

[0069] 7) Add the result obtained in the previous step to the minimum value, so that the original local extreme point position is the negative minimum value and the non-extreme point position is the positive minimum value.

[0070] 8) Perform ReLu activation operation on the result obtained in the previous step, so that the original local extreme point position is 0 and the non-extreme point position is a positive minimum value.

[0071] 9) Multiply the result obtained in the previous step by the (MUL) magnification factor, so that the minimum point position becomes 0 and the non-extreme point position becomes a negative smaller value.

[0072] 10) Add the result obtained in the previous step to the above step 5) to obtain the final heat map, which can directly obtain the local minimum value.

[0073] Therefore, the NMS operator is replaced by a general operator supported by the end-side device, thereby realizing the function of the NMS operator.

[0074] In some embodiments, a library of operator replacement algorithms can be pre-established, storing a variety of operator replacement algorithms for replacing operators that may not be supported by various end-side devices with basic operators that are generally supported by the end-side devices. Thus, all operator types in the AI ​​parsing model to be optimized can be filtered according to the range of operators supported by the end-side devices, and at least one operator replacement algorithm can be selected from the library of operator replacement algorithms. Based on this operator replacement method, unsupported operators can be replaced with basic operators.

[0075] Therefore, by replacing the operators of the AI ​​inference model according to the optimization requirements, the optimized deployment of the AI ​​inference model can be achieved without changing the hardware configuration of the end-side device (for example, replacing it with a hardware platform that supports all operators of the AI ​​inference model to be optimized).

[0076] Next, in S3040, the bandwidth of the AI ​​inference model can be compressed according to the optimization requirements so that the optimized AI inference model meets the output bandwidth limitation of the terminal device.

[0077] In some embodiments, a compression algorithm can be selected from a pre-prepared compression algorithm library to perform the compression on the output bandwidth of the AI ​​reasoning model according to the output bandwidth of the AI ​​reasoning model to be optimized and the output bandwidth limit of the end-side device. The selected compression algorithm may be, for example, a compression algorithm that can meet the output bandwidth limit of the end-side device and minimize loss. In addition, it is possible to determine whether the compression meets the optimization requirements based on the output bandwidth of the compressed AI reasoning model. If it does not meet the optimization requirements, other compression algorithms in the compression algorithm library are selected for compression. Compression algorithms may include, for example, but are not limited to, shift quantization compression algorithms, PACT (parameterized clipping activation) algorithms, floating-point compression algorithms, and modified CAE (Convolutional Auto-Encode) structure algorithms.

[0078] After completing the optimization of the AI ​​reasoning model through the above S3020 to S3040, the optimized AI parsing model, for example in json-similar format, can be reorganized to obtain the optimized AI reasoning model for deployment to the terminal device.

[0079] In addition, in S3050, the optimization results can be output. The optimization results may include one or more of the following: the degree of improvement in the inference efficiency of the optimized AI inference model, the operator replacement status, the degree of compression of the output bandwidth, precision loss, and latency. In addition, the optimization results may also include other information. For example, other information related to the optimization of the AI ​​inference model may be output based on the needs of the user of the end-side device. For example, after performing the structural reparameterization optimization in S3020, the degree of improvement in the inference efficiency of the optimized AI inference model may be output as the optimization result. The degree of improvement in inference efficiency may include, for example, the number of nodes in the optimized AI inference model. In addition, after performing the operator replacement in S3030, the operator replacement status (whether the replacement was successful or not) may be output as the optimization result. In addition, the operators replaced and / or the corresponding operator replacement algorithm may also be output as the optimization result. In addition, after performing the bandwidth compression in S3040, the degree of compression of the output bandwidth may be output as the optimization result. In addition, the specific compression algorithm used may also be output as the optimization result. In addition, the precision loss, latency, throughput, etc. of the optimized AI inference model may also be output. In addition, the optimization results are not limited to the examples listed above, and other optimization results related to the optimization of the AI ​​reasoning model can be output as needed.

[0080] It should be understood that the three optimization methods from S3020 to S3040 in method 3000 are not all necessary, and one or two of the optimization methods can be selected to optimize the AI ​​inference model according to specific optimization requirements.

[0081] In addition, in some embodiments, optimization options may also be provided to allow the user of the terminal device to independently select the optimization method to be performed. Next, this implementation method will be described in detail with reference to FIG4 .

[0082] FIG4 shows an exemplary flow chart of a method 4000 for optimizing an AI reasoning model according to another embodiment of the present disclosure. The method 4000 can be implemented, for example, by the electronic device 1000 shown in FIG1 , wherein each step can be implemented, for example, by configuring the processing circuit 1010 accordingly. In addition, steps S4010, S4020, S4030, S4040, and S4050 in the method 4000 can correspond, for example, to steps S3010 to S3050 in the method 3000, and the repeated contents will not be repeated here.

[0083] As shown in Figure 4, in S4010, the AI ​​reasoning model is parsed and optimization options for the AI ​​reasoning model are obtained. The optimization options can be, for example, one or more optimization methods selected by the user of the end-side device from the three optimization methods described above. In addition, in some embodiments, the information of the optimization options can also be included in the optimization requirements, and the user's optimization options for the AI ​​reasoning model to be deployed on the end-side device can be obtained by parsing the configuration file of the optimization requirements.

[0084] Next, at S4015, a determination is made as to whether the optimization options include a requirement for structural reparameterization. If the determination at S4015 is "yes," the process proceeds to S4020 to perform structural reparameterization of the AI ​​inference model. Otherwise, if the determination at S4015 is "no," the process proceeds to S4025 to determine the next optimization option.

[0085] Next, at S4025, a determination is made as to whether the optimization options include an optimization requirement for operator replacement. If the determination at S4025 is "yes," the process proceeds to S4030 to perform operator replacement for the AI ​​inference model. Otherwise, if the determination at S4025 is "no," the process proceeds to S4035 to determine the next optimization option.

[0086] Next, at S4035, a determination is made as to whether the optimization options include bandwidth compression optimization requirements. If the determination at S4035 is "yes," the process proceeds to S4040 to perform bandwidth compression on the AI ​​inference model, and then to S4040 to output the optimization results. Otherwise, if the determination at S4035 is "no," the process proceeds to S4040.

[0087] According to the method 4000 disclosed herein, users of the terminal device can select the desired optimization method to optimize the AI ​​inference model, which can increase the flexibility of the optimization and the adaptability to user needs.

[0088] In some embodiments, an optimization target can also be set. For example, the optimization target can be obtained from the user of the end-side device, and the optimization target can also be included in the optimization requirements, and the user's optimization target for the AI ​​reasoning model to be deployed on the end-side device can be obtained by parsing the configuration file of the optimization requirements. The optimization target can, for example, indicate one or more of the precision loss, IO bandwidth, throughput, latency, etc. of the optimized AI reasoning model. According to the optimization target, the AI ​​reasoning model optimization method of the present disclosure (for example, any one of methods 2000 to 4000) is executed. When the optimized AI reasoning model reaches the optimization target, the optimization is stopped and the optimized AI reasoning model and the optimization result are output. In addition, when the optimized AI reasoning model does not reach the optimization target, the AI ​​reasoning model optimization method can be iteratively executed, for example, one or more of the structural reparameterization method, the operator replacement algorithm, and the compression algorithm are modified until the optimized AI reasoning model reaches the optimization target.

[0089] In addition, in some embodiments, when no optimization target is set, or the set optimization target does not specify a fixed optimization target (fixed precision loss, IO bandwidth, throughput, delay, etc.), but when it is expected to achieve optimal optimization, the AI ​​reasoning model optimization method can also be iteratively executed until the optimal AI reasoning model is obtained. For example, a judgment criterion for the degree of optimization of the AI ​​reasoning model can be designed based on parameters such as precision loss, IO bandwidth, throughput, and delay, and the judgment criterion can be used to determine whether the optimized AI reasoning model is the optimal AI reasoning model.

[0090] In some embodiments, when an optimization target is set, information indicating whether the optimization is successful may also be output as an optimization result.

[0091] In some embodiments, the electronic device 1000 shown in Figure 1 can be mounted on an end-side device. Thus, the end-side device can use the method according to the present disclosure to optimize the AI ​​inference model to be deployed offline, without the need to retrain with user data, and without changing the hardware configuration of the end-side device. In other embodiments, the electronic device 1000 shown in Figure 1 can be configured on the server side, and the AI ​​inference model to be deployed to the end-side device can be optimized according to the hardware configuration of the end-side device, and the optimized AI inference model can be sent to the end-side device for deployment. Thus, optimized AI inference models that match their own hardware configurations can be provided to multiple end-side devices respectively, without using user data of each end-side device.

[0092] FIG. 5 illustrates an exemplary configuration of a computing device 500 capable of implementing embodiments according to the present invention.

[0093] Computing device 500 is an example of a hardware device to which the above-described aspects of the present invention can be applied. Computing device 500 can be any machine configured to perform processing and / or computations. Computing device 500 can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smartphone, an in-vehicle computer, or a combination thereof.

[0094] As shown in Figure 5, the computing device 500 may include one or more elements that can be connected or communicated with the bus 502 via one or more interfaces. The bus 502 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. The computing device 500 may include, for example, one or more processors 504, one or more input devices 506, and one or more output devices 508. The one or more processors 504 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). The processor 502 may, for example, correspond to the processor 1010 in Figure 1 and be configured to implement the functions of each unit of the apparatus for optimizing the AI ​​reasoning model of the present disclosure. The input device 506 may be any type of input device capable of inputting information to the computing device and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. Output device 508 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video / audio output terminals, a vibrator, and / or a printer.

[0095] The computing device 500 may also include or be connected to a non-transitory storage device 514, which may be any non-transitory storage device that can implement data storage and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a compact disk or any other optical medium, a cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code. The computing device 500 may also include a random access memory (RAM) 510 and a read-only memory (ROM) 512. The ROM 512 may store programs, utilities or processes to be executed in a non-volatile manner. The RAM 510 may provide volatile data storage and store instructions related to the operation of the computing device 500. The computing device 500 may also include a network / bus interface 516 coupled to a data link 518. The network / bus interface 516 can be any type of device or system capable of enabling communication with external devices and / or networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset (such as a Bluetooth™ device, an IEEE 802.11 device, a WiFi device, a WiMax device, a mobile cellular communication facility, etc.).

[0096] It should be understood that references to "an embodiment" or similar expressions throughout this specification mean that the specific features, structures, or characteristics described in connection with that embodiment are included in at least one specific embodiment of the present disclosure. Therefore, the appearance of phrases such as "in an embodiment of the present disclosure" or similar expressions throughout this specification does not necessarily refer to the same embodiment.

[0097] Those skilled in the art will appreciate that the present disclosure is implemented as a system, apparatus, method, or computer-readable storage medium (e.g., a non-transitory storage medium) as a computer program product. Thus, the present disclosure may be implemented in various forms, such as a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microprogram code, etc.), or a combination of software and hardware, hereinafter referred to as a "circuit," "module," or "system." Furthermore, the present disclosure may also be implemented as a computer program product in any tangible media form having computer-usable program code stored thereon.

[0098] The relevant description of the present disclosure is described with reference to the flowcharts and / or block diagrams of the systems, devices, methods and computer program products according to specific embodiments of the present disclosure. It is understood that each block in each flowchart and / or block diagram, as well as any combination of blocks in the flowchart and / or block diagram, can be implemented using computer program instructions. These computer program instructions can be executed by a machine composed of a processor of a general-purpose computer or a special computer or other programmable data processing device, and the instructions are processed by the computer or other programmable data processing device to implement the functions or operations described in the flowchart and / or block diagram.

[0099] The accompanying drawings show flowcharts and block diagrams of the architecture, functions and operations that can be implemented by the systems, devices, methods and computer program products according to various embodiments of the present disclosure. Therefore, each block in the flowchart or block diagram can represent a module, segment, or portion of program code, which includes one or more executable instructions to implement the specified logical function. It should also be noted that in certain other embodiments, the functions described by the blocks may not be performed in the order shown in the figures. For example, two blocks connected in the figures can actually be executed simultaneously, or in some cases can be executed in the opposite order of the icons depending on the functions involved. It should also be noted that each block of the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a system based on dedicated hardware, or by a combination of dedicated hardware and computer instructions to perform specific functions or operations.

[0100] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to marketed technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0101] Note that the technology disclosed in this specification can have the following configurations.

[0102] (1) An electronic device for optimizing an artificial intelligence (AI) inference model, the electronic device comprising:

[0103] a processing circuit, the processing circuit being configured to:

[0104] Extract the AI ​​inference model to be optimized;

[0105] Obtaining optimization requirements for the AI ​​inference model, the optimization requirements being related to one or more of the following configurations of an edge device where the AI ​​inference model will be deployed: a range of operators supported by the edge device, computing power of the edge device, and an output bandwidth limit of the edge device; and

[0106] According to the optimization requirements, the network structure of the AI ​​reasoning model is optimized to obtain an optimized AI reasoning model.

[0107] (2) The electronic device according to (1), wherein the processing circuit is further configured to: parse the AI ​​reasoning model to be optimized to determine a network structure of the AI ​​reasoning model,

[0108] Among them, according to the optimization requirements, the AI ​​reasoning model is structurally reparameterized to reduce the number of branches in the network structure of the AI ​​reasoning model, so that the optimized AI reasoning model matches the computing power of the end-side device.

[0109] (3) An electronic device according to (2), wherein the size of the convolution kernel in the network structure of the AI ​​reasoning model is further reduced by performing the structural reparameterization on the AI ​​reasoning model.

[0110] (4) An electronic device according to (3), wherein the network structure of the AI ​​reasoning model is optimized to a single-branch structure having a 3×3 convolution and activation function structure by performing the structural reparameterization on the AI ​​reasoning model.

[0111] (5) The electronic device according to (1), wherein the processing circuit is further configured to: parse the AI ​​reasoning model to be optimized to determine operators required to run the AI ​​reasoning model to be optimized,

[0112] Among them, according to the optimization requirements, operators within the operator range supported by the terminal side device are used to replace operators required to run the AI ​​reasoning model to be optimized.

[0113] (6) The electronic device according to (5), wherein the replaced operator includes an operator not supported by the end-side device.

[0114] (7) An electronic device according to (1), wherein the processing circuit is further configured to: compress the output bandwidth of the AI ​​inference model according to the optimization requirements so that the optimized AI inference model meets the output bandwidth limitation of the end-side device.

[0115] (8) An electronic device according to (7), wherein, based on the output bandwidth of the AI ​​inference model to be optimized and the output bandwidth limit of the terminal device, a compression algorithm is selected from a pre-prepared compression algorithm library to compress the output bandwidth of the AI ​​inference model.

[0116] (9) An electronic device according to (1), wherein the processing circuit is further configured to: output an optimization result, wherein the optimization result includes one or more of the following: the degree of improvement in the reasoning efficiency of the optimized AI reasoning model, the replacement status of operators, the degree of compression of the output bandwidth, the loss of precision, the delay, and the throughput rate.

[0117] (10) The electronic device according to (1), wherein the electronic device is mounted on the terminal side device.

[0118] (11) A method for optimizing an artificial intelligence (AI) inference model, comprising:

[0119] Extract the AI ​​inference model to be optimized;

[0120] Obtaining optimization requirements for the AI ​​inference model, the optimization requirements being related to one or more of the following configurations of an edge device where the AI ​​inference model will be deployed: a range of operators supported by the edge device, computing power of the edge device, and an output bandwidth limit of the edge device; and

[0121] According to the optimization requirements, the network structure of the AI ​​reasoning model is optimized to obtain an optimized AI reasoning model.

[0122] (12) The method according to (11), further comprising: parsing the AI ​​reasoning model to be optimized to determine the network structure of the AI ​​reasoning model,

[0123] Among them, according to the optimization requirements, the AI ​​reasoning model is structurally reparameterized to reduce the number of branches in the network structure of the AI ​​reasoning model, so that the optimized AI reasoning model matches the computing power of the end-side device.

[0124] (13) The method according to (12), wherein the size of the convolution kernel in the network structure of the AI ​​reasoning model is further reduced by performing the structural reparameterization on the AI ​​reasoning model.

[0125] (14) The method according to (13), wherein the network structure of the AI ​​reasoning model is optimized into a single-branch structure with a 3×3 convolution and activation function structure by performing the structural reparameterization on the AI ​​reasoning model.

[0126] (15) The method according to (11), further comprising: parsing the AI ​​reasoning model to be optimized to determine the operators required to run the AI ​​reasoning model to be optimized,

[0127] Among them, according to the optimization requirements, operators within the operator range supported by the terminal side device are used to replace operators required to run the AI ​​reasoning model to be optimized.

[0128] (16) The method according to (15), wherein the replaced operator includes an operator not supported by the end-side device.

[0129] (17) The method according to (11) further includes: compressing the output bandwidth of the AI ​​inference model according to the optimization requirements so that the optimized AI inference model meets the output bandwidth limitation of the terminal device.

[0130] (18) The method according to (17), wherein, based on the output bandwidth of the AI ​​inference model to be optimized and the output bandwidth limit of the terminal device, a compression algorithm is selected from a pre-prepared compression algorithm library to compress the output bandwidth of the AI ​​inference model.

[0131] (19) The method according to (11) further includes: outputting optimization results, wherein the optimization results include one or more of the following: the degree of improvement in the reasoning efficiency of the optimized AI reasoning model, the replacement status of operators, the degree of compression of the output bandwidth, the loss of precision, the delay, and the throughput rate.

[0132] (20) A computer-readable storage medium comprising executable instructions, which, when executed by an information processing device, causes the information processing device to execute the method according to any one of (11) to (19).

[0133] (21) A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the method described in any one of (11) to (19).

Claims

1. An electronic device for optimizing an artificial intelligence (AI) inference model, the electronic device comprising: A processing circuit, the processing circuit being configured to: Extract the AI ​​reasoning model to be optimized; Obtaining optimization requirements for the AI ​​reasoning model, the optimization requirements being related to one or more of the following configurations of an end-side device where the AI ​​reasoning model will be deployed: a range of operators supported by the end-side device, a computing capability of the end-side device, and an output bandwidth limit of the end-side device; and According to the optimization requirements, the network structure of the AI ​​reasoning model is optimized to obtain an optimized AI reasoning model.

2. The electronic device according to claim 1, wherein: The processing circuit is further configured to: parse the AI ​​reasoning model to be optimized to determine the network structure of the AI ​​reasoning model, Among them, according to the optimization requirements, the AI ​​reasoning model is structurally re-parameterized to reduce the number of branches in the network structure of the AI ​​reasoning model, so that the optimized AI reasoning model matches the computing power of the terminal device.

3. The electronic device according to claim 2, wherein: By performing the structural reparameterization on the AI ​​reasoning model, the size of the convolution kernel in the network structure of the AI ​​reasoning model is further reduced.

4. The electronic device according to claim 3, wherein: By performing the structural reparameterization on the AI ​​reasoning model, the network structure of the AI ​​reasoning model is optimized to a single-branch structure with a 3×3 convolution and activation function structure.

5. The electronic device according to claim 1, wherein: The processing circuit is further configured to: parse the AI ​​reasoning model to be optimized to determine operators required to run the AI ​​reasoning model to be optimized, Among them, according to the optimization requirements, operators within the operator range supported by the terminal side device are used to replace operators required to run the AI ​​reasoning model to be optimized.

6. The electronic device according to claim 5, wherein: The replaced operators include operators not supported by the terminal device.

7. The electronic device according to claim 1, wherein: The processing circuit is further configured to: compress the output bandwidth of the AI ​​reasoning model according to the optimization requirements so that the optimized AI reasoning model meets the output bandwidth limitation of the terminal side device.

8. The electronic device according to claim 7, wherein: According to the output bandwidth of the AI ​​reasoning model to be optimized and the output bandwidth limit of the terminal device, a compression algorithm is selected from a pre-prepared compression algorithm library to compress the output bandwidth of the AI ​​reasoning model.

9. The electronic device according to claim 1, wherein: The processing circuit is further configured to output optimization results, which include one or more of the following: the degree of improvement in reasoning efficiency of the optimized AI reasoning model, the replacement status of operators, the degree of compression of output bandwidth, precision loss, delay, and throughput.

10. The electronic device according to claim 1, wherein: The electronic device is mounted on the terminal side device.

11. A method for optimizing an artificial intelligence (AI) inference model, comprising: Extract the AI ​​reasoning model to be optimized; Obtaining optimization requirements for the AI ​​reasoning model, the optimization requirements being related to one or more of the following configurations of an end-side device where the AI ​​reasoning model will be deployed: a range of operators supported by the end-side device, a computing capability of the end-side device, and an output bandwidth limit of the end-side device; and According to the optimization requirements, the network structure of the AI ​​reasoning model is optimized to obtain an optimized AI reasoning model.

12. The method according to claim 11, further comprising: parsing the AI ​​reasoning model to be optimized to determine the network structure of the AI ​​reasoning model, Among them, according to the optimization requirements, the AI ​​reasoning model is structurally re-parameterized to reduce the number of branches in the network structure of the AI ​​reasoning model, so that the optimized AI reasoning model matches the computing power of the terminal device.

13. The method according to claim 12, wherein: By performing the structural reparameterization on the AI ​​reasoning model, the size of the convolution kernel in the network structure of the AI ​​reasoning model is further reduced.

14. The method according to claim 13, wherein: By performing the structural reparameterization on the AI ​​reasoning model, the network structure of the AI ​​reasoning model is optimized to a single-branch structure with a 3×3 convolution and activation function structure.

15. The method according to claim 11, further comprising: parsing the AI ​​reasoning model to be optimized to determine operators required to run the AI ​​reasoning model to be optimized, Among them, according to the optimization requirements, operators within the operator range supported by the terminal side device are used to replace operators required to run the AI ​​reasoning model to be optimized.

16. The method according to claim 15, wherein: The replaced operators include operators not supported by the terminal device.

17. The method according to claim 11, further comprising: According to the optimization requirements, the output bandwidth of the AI ​​reasoning model is compressed so that the optimized AI reasoning model meets the output bandwidth limitation of the terminal device.

18. The method according to claim 17, wherein: According to the output bandwidth of the AI ​​reasoning model to be optimized and the output bandwidth limit of the terminal device, a compression algorithm is selected from a pre-prepared compression algorithm library to compress the output bandwidth of the AI ​​reasoning model.

19. The method according to claim 11, further comprising: Output optimization results, which include one or more of the following: the degree of improvement in reasoning efficiency of the optimized AI reasoning model, the replacement status of operators, the degree of compression of output bandwidth, precision loss, delay, and throughput. 20 . A computer-readable storage medium comprising executable instructions, which, when executed by an information processing device, causes the information processing device to execute the method according to any one of claims 11 to 19.

21. A computer program product, comprising a computer program, which, when executed by a processor, causes the processor to perform the method according to any one of claims 11 to 19.

Images