Inference acceleration methods, inference acceleration devices and storage media

By compressing and parallelizing the neural network model, and utilizing fixed-point optimization and SIMD optimization modules, the problems of high computational load and high power consumption of neural network models on mobile edge devices are solved, achieving faster computing speed and lower power consumption, and making it suitable for various network models.

CN114781616BActive Publication Date: 2024-11-08BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210460454.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-24
Publication Date
2024-11-08
Estimated Expiration
2042-04-24

AI Technical Summary

Technical Problem

When running neural network models on mobile edge devices, existing technologies suffer from high computational load, high power consumption, and difficulty in running for extended periods, especially when using CPUs and GPUs for inference acceleration, which consumes a lot of resources.

Method used

By compressing the neural network model to be accelerated, parallel acceleration processing is performed using fixed-point optimization and SIMD optimization modules, including fixed-point bit-width optimization and SIMD instruction conversion, thereby optimizing the structure and computational efficiency of the neural network model.

Benefits of technology

It achieves faster computing speeds and lower power consumption on mobile edge devices. The acceleration scheme is applicable to both compressible and incompressible network models, improving the universality of inference methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114781616B_ABST
    Figure CN114781616B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an inference acceleration method, an inference acceleration device and a storage medium. The method comprises: obtaining first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; inputting the first model information and the second model information into an inference acceleration system, using a fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model respectively, and determining a third network model and fixed-point configuration information of the third network model; using a single instruction multiple data (SIMD) optimization module of the inference acceleration system to optimize and accelerate the third network model based on SIMD instructions, and obtaining SIMD configuration information of the accelerated target network model; and determining target model configuration information of the target network model according to the fixed-point configuration information SIMD configuration information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to an inference acceleration method, an inference acceleration device and a storage medium. Background Art

[0002] With the development of computer technology, neural network models have been successfully applied to many fields such as image recognition processing and autonomous driving. As application requirements continue to increase, the number of network layers in neural network models increases. The increase in network layers leads to an increasing depth of the neural network model, a significant increase in computational complexity, and a significant reduction in processing efficiency.

[0003] In the related technology, the reasoning of neural network models is usually accelerated through hardware such as central processing unit (CPU), image processing unit (GPU), neural network processor (NPU) and digital signal processor (DSP); however, using CPU and GPU to run neural network models on mobile edge devices such as mobile phones requires more terminal resources and consumes higher power, and cannot achieve long-term operation (for example, 24 hours). Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an inference acceleration method, an inference acceleration device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for accelerating reasoning, including:

[0006] Acquire first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model;

[0007] Inputting the first model information and the second model information into an inference acceleration system, and using a fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information;

[0008] Determine a third network model and fixed-point configuration information of the third network model based on the first network model and the second network model after fixed-point optimization processing;

[0009] Utilizing a single instruction multiple data (SIMD) optimization module of the inference acceleration system, optimizing and accelerating the third network model based on SIMD instructions, and obtaining SIMD configuration information of the accelerated target network model;

[0010] The target model configuration information of the target network model is determined according to the fixed-point configuration information and the SIMD configuration information.

[0011] Optionally, the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel;

[0012] The fixed-point optimization module of the inference acceleration system is used to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information, respectively, including:

[0013] Using the first fixed-point optimization submodule, based on the first model information and the second model information, randomly determine a first fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0014] Utilizing the second fixed-point optimization submodule, sensitivity processing is performed on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, the second fixed-point bit width corresponding to each layer in the first network model and the second network model is determined.

[0015] Optionally, the determining the third network model and the fixed-point configuration information of the third network model based on the first network model and the second network model after the fixed-point optimization processing includes:

[0016] According to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, the network parameters in the first network model and the second network model are subjected to first fixed-point processing to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model;

[0017] Based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model, a third network model and fixed-point configuration information corresponding to the third network model are determined.

[0018] Optionally, the method further comprises:

[0019] The fixed-point configuration information corresponding to the third network model is fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule; the fixed-point configuration information is used by the first fixed-point optimization submodule and the second fixed-point optimization submodule to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

[0020] Optionally, the SIMD optimization module comprises: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel;

[0021] The single instruction multiple data (SIMD) optimization module of the inference acceleration system is used to optimize and accelerate the third network model based on SIMD instructions to obtain SIMD configuration information of the accelerated target network model, including:

[0022] Using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model;

[0023] Utilizing the second SIMD optimization submodule, randomly extracting a plurality of operation instructions to be converted from the third network model, converting the plurality of operation instructions into a plurality of SIMD instructions respectively, to obtain a converted fifth network model; and determining the performance parameters and running time of the fourth network model and the fifth network model respectively;

[0024] Based on the performance parameters and running time of the fourth network model and the fifth network model, the target network model and SIMD configuration information of the target network model are determined.

[0025] Optionally, the method comprises:

[0026] The SIMD configuration information of the target network model is fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

[0027] Optionally, the acquiring first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model includes:

[0028] Acquire a first network model to be accelerated and the first model information of the first network model;

[0029] The first network model is pruned to obtain the second network model, and the second model information is obtained based on the second network model.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided an inference acceleration device, including:

[0031] An acquisition module, used to acquire first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model;

[0032] A first optimization module, configured to input the first model information and the second model information into an inference acceleration system, and use a fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information; and determine a third network model and fixed-point configuration information of the third network model based on the first network model and the second network model after the fixed-point optimization processing;

[0033] A second optimization module is used to optimize and accelerate the third network model based on SIMD instructions by using the single instruction multiple data (SIMD) optimization module of the inference acceleration system to obtain SIMD configuration information of the accelerated target network model;

[0034] A determination module is used to determine the target model configuration information of the target network model according to the fixed-point configuration information and the SIMD configuration information.

[0035] Optionally, the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel;

[0036] The first optimization module is used to:

[0037] Using the first fixed-point optimization submodule, based on the first model information and the second model information, randomly determine a first fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0038] Utilizing the second fixed-point optimization submodule, sensitivity processing is performed on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, the second fixed-point bit width corresponding to each layer in the first network model and the second network model is determined.

[0039] Optionally, the first optimization module is used to:

[0040] According to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, the network parameters in the first network model and the second network model are subjected to first fixed-point processing to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model;

[0041] Based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model, a third network model and fixed-point configuration information corresponding to the third network model are determined.

[0042] Optionally, the first optimization module is used to:

[0043] The fixed-point configuration information corresponding to the third network model is fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule; the fixed-point configuration information is used by the first fixed-point optimization submodule and the second fixed-point optimization submodule to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

[0044] Optionally, the SIMD optimization module comprises: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel;

[0045] The second optimization module is used for:

[0046] Using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model;

[0047] Utilizing the second SIMD optimization submodule, randomly extracting a plurality of operation instructions to be converted from the third network model, converting the plurality of operation instructions into a plurality of SIMD instructions respectively, to obtain a converted fifth network model; and determining the performance parameters and running time of the fourth network model and the fifth network model respectively;

[0048] Based on the performance parameters and running time of the fourth network model and the fifth network model, the target network model and SIMD configuration information of the target network model are determined.

[0049] Optionally, the second optimization module is used to:

[0050] The SIMD configuration information of the target network model is fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

[0051] Optionally, the acquisition module is used to:

[0052] Acquire a first network model to be accelerated and the first model information of the first network model;

[0053] The first network model is pruned to obtain the second network model, and the second model information is obtained based on the second network model.

[0054] According to a third aspect of an embodiment of the present disclosure, there is provided an inference acceleration device, including:

[0055] processor;

[0056] a memory for storing executable instructions;

[0057] Wherein, the processor is configured to: when executing the executable instructions stored in the memory, implement the steps in the reasoning acceleration method described in the first aspect of the embodiment of the present disclosure.

[0058] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an inference acceleration device, the inference acceleration device is enabled to perform the steps in the inference acceleration method described in the first aspect of an embodiment of the present disclosure.

[0059] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0060] The disclosed embodiment compresses the first network model to be accelerated to obtain a compressed second network model, and accelerates the first network model and the second network model in parallel, thereby accelerating the first network model from the aspect of reducing the complexity of the network structure; utilizes the fixed-point optimization module and SIMD optimization model in the inference acceleration system to perform fixed-point optimization and SIMD optimization on the first network model and the second network model, respectively, so that the operation speed of the accelerated target network model is faster and the power consumption is lower.

[0061] In addition, the original first network model and the compressed second network model are accelerated in parallel, so that a better scheme is selected from the acceleration scheme of the first network model and the acceleration scheme of the second network model. This scheme can be applicable to both compressible network models and incompressible network models, thereby improving the universal applicability of the accelerated inference method.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0064] Figure 1 It is a processing diagram of a CPU-based compilation collaborative optimization algorithm provided by related technologies.

[0065] Figure 2 is a flowchart of a method for accelerating reasoning according to an exemplary embodiment. Figure 1 .

[0066] Figure 3 is a flowchart of a method for accelerating reasoning according to an exemplary embodiment. Figure 2 .

[0067] Figure 4 The figure is a schematic diagram of inference acceleration processing of a network model according to an exemplary embodiment.

[0068] Figure 5 It is a schematic diagram of a framework of a fixed-point optimization module according to an exemplary embodiment.

[0069] Figure 6 It is a structural diagram of a SIMD optimization module according to an exemplary embodiment.

[0070] Figure 7 The figure is a schematic diagram of the structure of an inference acceleration device according to an exemplary embodiment.

[0071] Figure 8 The figure is a block diagram of an inference acceleration device according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0073] In related technologies, neural network models are usually compiled and optimized on NPU, CPU and GPU, such as Figure 1 As shown, Figure 1 This is a processing diagram of a CPU-based compilation collaborative optimization algorithm provided by related technologies. By pruning the algorithm, the number of numerical operations in the algorithm is reduced, and then the number of instructions required to read data is reduced, and finally the computing load of each thread and each algorithm block is balanced.

[0074] However, this CPU-based optimization of algorithm operation does not take into account the power consumption of mobile edge devices, and does not automatically complete the compilation and collaborative optimization process of the entire algorithm.

[0075] The present disclosure provides a method for accelerating reasoning. Figure 2 As shown, Figure 2 is a flowchart of a method for accelerating reasoning according to an exemplary embodiment. Figure 1 The method comprises:

[0076] Step S101, obtaining first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model;

[0077] Step S102, inputting the first model information and the second model information into an inference acceleration system, and using a fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information;

[0078] Step S103, determining a third network model and fixed-point configuration information of the third network model based on the first network model and the second network model after fixed-point optimization processing;

[0079] Step S104, using the single instruction multiple data (SIMD) optimization module of the inference acceleration system to optimize and accelerate the third network model based on SIMD instructions to obtain SIMD configuration information of the accelerated target network model;

[0080] Step S105 , determining target model configuration information of the target network model according to the fixed-point configuration information and the SIMD configuration information.

[0081] It should be noted that, considering the advantages of DSP such as low power consumption and high computing speed, by using DSP to run neural network models on mobile edge devices, the power consumption and time consumption of mobile edge devices can be reduced. However, the compilation coordination solutions in related technologies are usually designed based on NPU, CPU and GPU. Compared with NPU, CPU and GPU, DSP has fewer computing resources, so it is necessary to optimize and accelerate the neural network model based on the characteristics of DSP.

[0082] In step S101, a first network model to be accelerated and first model information of the first network model are obtained, the first network model is compressed to obtain the second network model; and second model information of the second network model is obtained.

[0083] Here, the first network model refers to a pre-trained neural network model, and the neural network model can be a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), etc. The specific one can be selected according to the actual situation, and the embodiments of the present disclosure are not limited to this.

[0084] The first model information includes at least: a network layer type and network parameters of each network layer constituting the first network model.

[0085] Each convolutional layer of the first network model may be compressed according to a predetermined compression ratio to obtain a second network model;

[0086] The compression processing method is specifically as follows: globally sorting each convolution channel in each convolution layer in the first network model according to the batch normalization (BN) weight value, closing a batch of convolution channels with smaller BN weight values ​​in the first network model, thereby obtaining a second network model. It can be understood that closing convolution channels with small BN weight values ​​has little effect on the accuracy of the final model, but can greatly reduce the complexity of the network model, reduce the amount of calculation, and improve the calculation speed.

[0087] It should be noted that the network model usually includes a BN layer, which is used to normalize the output values ​​of each convolution channel in the associated network layer. The BN weight value of each convolution channel in the network layer can be obtained through the BN layer; the BN weight value can directly reflect the importance of the convolution channel.

[0088] The second model information includes at least: a network layer type and network parameters of each network layer constituting the second network model.

[0089] It should be noted that the embodiment of the present disclosure compresses the first network model to be accelerated to obtain a compressed second network model, and accelerates the first network model and the second network model in parallel. On the one hand, the first network model is accelerated by reducing the complexity of the network structure; on the other hand, through parallel acceleration, a better scheme is selected from the acceleration scheme of the first network model and the acceleration scheme of the second network model, thereby improving the scope of application of the accelerated reasoning method.

[0090] In step S102, the first model information and the second model information are input into the inference acceleration system, and the inference acceleration system performs inference acceleration on the first network model and the second network model based on the first model information and the second model information, respectively.

[0091] Here, the inference acceleration system includes: a fixed-point optimization module and a SIMD optimization module.

[0092] The fixed-point optimization module can be used to perform fixed-point optimization processing on the first network model and the second network model respectively based on the first model information, the second model information and the fixed-point bit width hardware resources of the operation unit to obtain the fixed-point bit widths of the first network model and the second network model.

[0093] It should be noted that the operation unit may be a computing unit that supports fixed-point operations, and the operation unit described in the embodiment of the present disclosure may be a DSP. In order to give full play to the comprehensive optimization of DSP in terms of low power consumption and high operation speed, the fixed-point bit width resources of the operation unit are usually limited. In order to achieve lower power consumption, the smallest possible fixed-point bit width resources are often selected, such as 8 bits or 4 bits or even less fixed-point bit width resources to achieve fixed-point operations. However, the small amount of fixed-point bit width resources will affect the operation speed; therefore, when accelerating the inference of the neural network model, adaptive fixed-point optimization processing can be performed on the neural network model to be accelerated, so as to take into account both power consumption and operation speed.

[0094] In the embodiment of the present disclosure, the positioning optimization module determines the network layer type of each network layer in the first network model based on the first model information, and determines the fixed-point width corresponding to each network layer based on the network layer type of each network layer;

[0095] Based on the second model information, the network layer type of each network layer in the second network model is determined, and based on the network layer type of each network layer, the fixed-point bit width corresponding to each network layer is determined.

[0096] Here, the amount of computation involved in each network layer in the network model can be sorted according to the network layer type, and the fixed-point bit width corresponding to each network layer in the network model can be determined according to the sorting result. It can be understood that if the amount of computation of a certain network layer in the network model is large, the fixed-point bit width corresponding to the network layer can be set larger, such as 32 bits, so as to increase the computing speed; if the amount of computation of a certain network layer in the network model is small, the fixed-point bit width corresponding to the network layer can be set smaller, such as 4 bits, so as to reduce power consumption.

[0097] In step S103, after determining the fixed-point bit widths of the first network model and the second network model, the network parameters in the first network model and the network parameters in the second network model may be fixed-point processed based on the fixed-point bit widths of the first network model and the second network model, respectively, to obtain the fixed-point parameters of the first network model and the fixed-point parameters of the second network model;

[0098] A third network model with better fixed-point effect and the fixed-point configuration information of the third network model may be determined according to the fixed-point parameters of the first network model and the fixed-point parameters of the second network model.

[0099] Here, the fixed-point configuration information may include the fixed-point bit width and fixed-point parameters corresponding to each network layer in the third network model.

[0100] It should be noted that the network parameters of the neural network model are usually floating-point numbers. The algorithm model based on floating-point numbers can be quickly trained and verified on a general-purpose processor development platform; the verified algorithm model can be implemented on a floating-point hardware platform, such as a floating-point DSP; but if you want to achieve faster speed and lower power consumption, you can convert the algorithm model based on floating-point numbers into an algorithm model based on fixed-point numbers.

[0101] In step S104, a SIMD optimization module may be used to randomly extract at least one operation instruction in the third network model, and convert the operation instruction into a SIMD instruction to obtain a converted target network model and SIMD configuration parameters of the target network model.

[0102] Here, the SIMD configuration parameters at least include the operation instructions to be converted, SIMD instructions, and network parameters involved in the operation instructions.

[0103] It can be understood that SIMD technology is a technology that uses a controller to control multiple processors or multiple processing elements, and simultaneously performs the same operation on each of a group of data, thereby achieving spatial parallelism; the key to the SIMD technology is to simultaneously execute multiple calculation operations in a single instruction to increase the throughput of the processor.

[0104] In the disclosed embodiment, a SIMD optimization module is used to randomly extract at least one operation instruction from the input third network model, determine the fixed-point parameters in the third network model that participate in the operation instruction; and define at least one vector data, wherein one vector data stores multiple fixed-point parameters that participate in one operation instruction; the operation instruction in the third network model that operates on the multiple fixed-point parameters is converted into a SIMD instruction that operates on the vector data to obtain a converted target network model. Thus, in the converted target network model, by executing the SIMD instruction, the fixed-point parameters that participate in the operation instruction can be calculated in parallel in the form of vector data, thereby reducing the number of operations of the target network model and improving the operation efficiency.

[0105] In step S105, after obtaining the target network model output by the SIMD optimization submodule and the SIMD configuration information of the target network model, the target model configuration information of the accelerated target network model may be determined based on the fixed-point configuration information and the SIMD configuration information.

[0106] Based on the target model configuration information, a target network model that is accelerated from the first network model can be constructed, so that corresponding information processing can be performed using the target network model to improve information processing efficiency.

[0107] It should be noted that after determining the target model configuration information, the target model configuration information can be sent to the mobile edge device, so that the mobile edge device can run the accelerated target network model based on the target model configuration information, thereby reducing the operating power consumption of the mobile edge device and improving the operating speed of the mobile edge device.

[0108] The disclosed embodiment compresses the first network model to be accelerated to obtain a compressed second network model, and accelerates the first network model and the second network model in parallel, thereby accelerating the first network model from the aspect of reducing the complexity of the network structure; utilizes the fixed-point optimization module and SIMD optimization model in the inference acceleration system to perform fixed-point optimization and SIMD optimization on the first network model and the second network model, respectively, so that the operation speed of the accelerated target network model is faster and the power consumption is lower.

[0109] In addition, the original first network model and the compressed second network model are accelerated in parallel, so that a better scheme is selected from the acceleration scheme of the first network model and the acceleration scheme of the second network model. This scheme can be applicable to both compressible network models and incompressible network models, thereby improving the universal applicability of the accelerated inference method.

[0110] In some embodiments, the method further comprises:

[0111] A second fixed-point processing is performed on the network parameters of the first network model and the second network model respectively; the fixed-point bit width of the second fixed-point processing is greater than the fixed-point bit width of the first fixed-point processing.

[0112] The step of inputting the first model information and the second model information into the inference acceleration system includes:

[0113] The first model information and the second model information after the second fixed-point processing are input into the inference acceleration system.

[0114] In an embodiment of the present disclosure, before inputting the first model information and the second model information into the inference acceleration system, the first network model and the second network model may be roughly fixed-point processed (ie, second fixed-point processed).

[0115] Here, the first network model and the second network model can be subjected to a second fixed-point processing according to the maximum fixed-point bit width supported by the computing unit; it can be understood that by performing a second fixed-point processing on the first network model and the second network model, the computing speed of the first network model and the second network model can be greatly improved.

[0116] The first model information and the second model information after the second fixed-point processing are input into the inference acceleration system, and the first network model and the second network model after the second fixed-point processing are subjected to fixed-point optimization processing based on the first model information and the second model information by using a fixed-point optimization module, thereby narrowing the range of the fixed-point bit width selected by the fixed-point optimization processing (i.e., the fixed-point bit width of the precise fixed-point processing) and improving the optimization efficiency.

[0117] Optionally, the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel;

[0118] The fixed-point optimization module of the inference acceleration system is used to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information, respectively, including:

[0119] Using the first fixed-point optimization submodule, based on the first model information and the second model information, randomly determine a first fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0120] Utilizing the second fixed-point optimization submodule, sensitivity processing is performed on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, the second fixed-point bit width corresponding to each layer in the first network model and the second network model is determined.

[0121] In the embodiment of the present disclosure, the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule; the first fixed-point optimization submodule and the second fixed-point optimization submodule are used to perform fixed-point optimization processing in parallel.

[0122] The first fixed-point optimization submodule may determine the network layer type of each network layer in the neural network model corresponding to the model information according to the input model information, and randomly determine the first fixed-point bit width corresponding to each network layer in the neural network model according to the network layer type;

[0123] The second fixed-point optimization submodule can determine the network layer type and network parameters of each network layer in the neural network model corresponding to the model information according to the input model information; and perform sensitivity processing on the network parameters contained in each network layer in the neural network model to determine the bit width sensitivity of each network layer in the neural network model to the fixed-point bit width;

[0124] The bit width sensitivity can be used to assist in determining the fixed-point bit width corresponding to the network layer. It is understandable that if the bit width sensitivity of the network parameters in a certain network layer in the neural network model is relatively large, the fixed-point bit width of the network layer can be set relatively large, such as 16 bits, 8 bits, etc. If the bit width sensitivity of the network parameters in a certain network layer in the neural network model is relatively small, the fixed-point bit width of the network layer can be set relatively small, such as 4 bits, 2 bits, etc.

[0125] Here, the sensitivity processing method can be selected according to the actual situation. For example, the Hessian matrix of the neural network model can be obtained by performing a second-order analysis (Hessian) on the neural network model, and the bit width sensitivity of each network layer in the neural network model can be determined based on the trace of the Hessian matrix. This disclosure embodiment is not limited to this.

[0126] After determining the bit width sensitivity of each network layer in the neural network model, the second fixed-point optimization submodule randomly sets a second fixed-point bit width for each network layer in the neural network model based on the bit width sensitivity corresponding to each network layer.

[0127] In the disclosed embodiment, after the first model information is input into the fixed-point optimization module, the first fixed-point optimization submodule is used to randomly determine the first fixed-point bit width corresponding to each network layer in the first network model based on the first model information; the second fixed-point optimization submodule is used to perform sensitivity processing on the first model information to obtain the bit width sensitivity of each network layer in the first network model, and the second fixed-point bit width of each network layer in the first network model is randomly determined based on the bit width sensitivity of each network layer.

[0128] After the second model information is input into the fixed-point optimization module, the first fixed-point optimization submodule is used to randomly determine the first fixed-point bit width corresponding to each network layer in the second network model based on the second model information; the second fixed-point optimization submodule is used to perform sensitivity processing on the second model information to obtain the bit width sensitivity of each network layer in the second network model, and based on the bit width sensitivity of each network layer, the second fixed-point bit width of each network layer in the second network model is randomly determined.

[0129] The disclosed embodiment utilizes a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel, and based on the first model information and the second model information, provides two different fixed-point bit width solutions for the first network model and the second network model, respectively, to improve the fixed-point optimization efficiency.

[0130] Optionally, the determining the third network model and the fixed-point configuration information of the third network model based on the first network model and the second network model after the fixed-point optimization processing includes:

[0131] According to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, the network parameters in the first network model and the second network model are subjected to first fixed-point processing to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model;

[0132] Based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model, a third network model and fixed-point configuration information corresponding to the third network model are determined.

[0133] In an embodiment of the present disclosure, after obtaining the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, a first fixed-point processing is performed on the network parameters of the first network model based on the first fixed-point bit width and the second fixed-point bit width of the first network model to obtain the first fixed-point parameters and the second fixed-point parameters of the first network model; a second fixed-point processing is performed on the network parameters of the second network model based on the first fixed-point bit width and the second fixed-point bit width of the second network model to obtain the first fixed-point parameters and the second fixed-point parameters of the second network model.

[0134] It should be noted that the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model are determined by the first fixed-point bit width and the second fixed-point bit width of the first network model and the second network model, and the first fixed-point bit width and the second fixed-point bit width are randomly set based on the fixed-point optimization module according to the network layer type and / or the bit width sensitivity of the network parameters of the first network model and the second network model. If the first fixed-point bit width and the second fixed-point bit width of the first network model and the second network model are inaccurate, the integer bit width and the decimal bit width of the fixed-point data will be magnified in the second fixed-point processing, making the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model inaccurate, affecting the calculation accuracy of the neural network model.

[0135] Therefore, the embodiments of the present disclosure can determine the processing accuracy corresponding to the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model based on the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model, and based on the processing accuracy, determine the third network model with the best processing accuracy and the fixed-point configuration information of the third network model from the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model.

[0136] Here, the fixed-point configuration information at least includes: a fixed-point bit width and fixed-point parameters.

[0137] It can be understood that, according to the processing accuracy of multiple fixed-point solutions determined by the first fixed-point optimization submodule and the second fixed-point optimization submodule based on the first network model and the second network model, a fixed-point solution with the best processing accuracy is determined from the multiple fixed-point solutions. The fixed-point solution with the best processing accuracy is determined as the fixed-point solution of the third network model.

[0138] In some embodiments, a third network model and fixed-point configuration information of the third network model that meet preset accuracy conditions can be determined based on the processing accuracy corresponding to the first fixed-point parameters and the second fixed-point parameters based on the first network model and the second network model.

[0139] In the embodiment of the present disclosure, the preset accuracy condition may be: the processing accuracy corresponding to the fixed-point parameter is greater than a preset accuracy threshold.

[0140] Whether the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model meet the preset accuracy conditions can be determined based on the comparison result of the processing accuracy corresponding to the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model with the preset accuracy threshold.

[0141] If the processing accuracies corresponding to the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model are both less than the preset accuracy threshold, it is determined that the processing accuracies corresponding to the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model do not meet the preset accuracy conditions; the first fixed-point optimization submodule and the second fixed-point optimization submodule can be used to re-determine the first fixed-point bit width and the second fixed-point bit width of the first network model and the second network model.

[0142] If at least one of the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model meets the preset accuracy condition, the network model that meets the preset accuracy condition is determined as the third network model, and the fixed-point parameter that meets the preset accuracy condition is determined as the fixed-point configuration information corresponding to the third network model.

[0143] It should be noted that if at least two of the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model meet the preset accuracy condition, a third network model and fixed-point configuration information with greater processing accuracy are determined.

[0144] The disclosed embodiment performs first fixed-point processing on the network parameters of the first network model and the second network model according to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, so as to obtain the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model; determines the third network model with the best processing accuracy and the fixed-point configuration information corresponding to the third network model according to the processing accuracy corresponding to the first fixed-point parameters and the second fixed-point parameters of the first network model and the second network model, which can not only improve the computing speed of the network model and reduce power consumption, but also reduce manual intervention in the fixed-point optimization and processing process, and improve the efficiency of the fixed-point optimization and processing.

[0145] Optionally, the method further comprises:

[0146] The fixed-point configuration information corresponding to the third network model is fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule; the fixed-point configuration information is used by the first fixed-point optimization submodule and the second fixed-point optimization submodule to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

[0147] In an embodiment of the present disclosure, after determining the third network model and the fixed-point position information corresponding to the third network model, the fixed-point configuration information can be fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule, so that the first fixed-point optimization submodule and the second fixed-point optimization submodule can determine the first fixed-point bit width and the second fixed-point bit width of the first network model and the second network model next time based on the fed-back fixed-point configuration information, as well as the network layer type and / or bit width sensitivity of the network parameters of the first network model and the second network model.

[0148] It should be noted that the embodiment of the present disclosure assists the first fixed-point optimization submodule and the second fixed-point optimization submodule in determining the first fixed-point bit width and the second fixed-point bit width of the first network model and the second network model by feeding back the fixed-point configuration information of the third network model; thereby, in the process of cyclic feedback, the fixed-point bit width and fixed-point parameters for the best fixed-point effect of the first network model and the second network model are gradually determined.

[0149] Optionally, the SIMD optimization module comprises: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel;

[0150] The single instruction multiple data (SIMD) optimization module of the inference acceleration system is used to optimize and accelerate the third network model based on SIMD instructions to obtain SIMD configuration information of the accelerated target network model, including:

[0151] Using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model;

[0152] Utilizing the second SIMD optimization submodule, randomly extracting a plurality of operation instructions to be converted from the third network model, converting the plurality of operation instructions into a plurality of SIMD instructions respectively, to obtain a converted fifth network model; and determining the performance parameters and running time of the fourth network model and the fifth network model respectively;

[0153] Based on the performance parameters and running time of the fourth network model and the fifth network model, the target network model and SIMD configuration information of the target network model are determined.

[0154] In the disclosed embodiment, the SIMD optimization module includes: a first SIMD optimization submodule and a second SIMD optimization submodule. The SIMD optimization is performed in parallel using the first SIMD optimization submodule and the second SIMD optimization submodule.

[0155] The first SIMD optimization submodule may randomly extract a calculation instruction to be converted from the input third network model, convert the calculation instruction into a SIMD instruction, and obtain a fourth network model after SIMD optimization;

[0156] The second SIMD optimization submodule can randomly extract a plurality of operation instructions to be converted from the input third network model, and convert the plurality of operation instructions into a plurality of SIMD instructions respectively, so as to obtain a fifth network model after SIMD optimization;

[0157] By determining the performance parameters and running time of the fourth network model and the fifth network model, determining a target network model with better performance parameters and shorter running time and SIMD configuration information corresponding to the target network model;

[0158] Here, the SIMD configuration information at least includes: the operation instruction to be converted, the SIMD instruction and the network parameters involved in the operation instruction.

[0159] The disclosed embodiment utilizes a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel to provide two different SIMD conversion schemes for the third network model respectively, and determines a better SIMD conversion scheme based on the performance parameters and running time of the two different SIMD conversion schemes.

[0160] In some embodiments, a target network model that meets preset optimization conditions and SIMD configuration information of the target network model may be determined according to the performance parameters and running time of the fourth network model and the fifth network model.

[0161] In the embodiment of the present disclosure, the preset optimization condition may be: the performance parameter is better than a preset performance parameter threshold, and the running time is less than a preset time threshold.

[0162] Whether the fourth network model and the fifth network model meet the preset optimization conditions can be determined based on a first comparison result of the performance parameters of the fourth network model and the fifth network model and a preset performance parameter threshold, and a second comparison result of the running time of the fourth network model and the fifth network model and a preset time threshold.

[0163] If the fourth network model and the fifth network model do not meet the preset optimization conditions, the first SIMD optimization submodule and the second SIMD optimization submodule may be used to re-determine the fourth network model and the fifth network model.

[0164] If at least one of the fourth network model and the fifth network model meets the preset optimization condition, the network model that meets the preset optimization condition is determined as the target network model.

[0165] It should be noted that if the fourth network model and the fifth network model both meet the preset optimization condition, a target network model with better performance parameters is selected from the fourth network model and the fifth network model.

[0166] The disclosed embodiment utilizes the first SIMD optimization submodule and the second SIMD optimization submodule to perform different SIMD optimizations on the third network model in parallel to obtain two different network models; a better network model is determined based on the performance parameters and running time corresponding to the two different network models; on the one hand, the SIMD optimization efficiency can be improved, and the computing efficiency of the network model can be improved; on the other hand, manual intervention in the SIMD optimization process can be reduced, and the degree of automation of the SIMD optimization process can be improved.

[0167] Optionally, the method comprises:

[0168] The SIMD configuration information of the target network model is fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

[0169] In an embodiment of the present disclosure, after determining the SIMD configuration information of the target network model, the SIMD configuration information can be fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, so that the first SIMD optimization submodule and the second SIMD optimization submodule can determine the operation instructions within the network model to be converted based on the fed-back SIMD configuration information.

[0170] It should be noted that the embodiment of the present disclosure assists the first SIMD optimization submodule and the second SIMD optimization submodule in determining the fourth network model and the fifth network model by feeding back the SIMD configuration information of the target network model, thereby gradually determining the target network model with the best SIMD optimization effect in the process of cyclic feedback.

[0171] Optionally, the acquiring first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model includes:

[0172] Acquire a first network model to be accelerated and the first model information of the first network model;

[0173] The first network model is pruned to obtain the second network model, and the second model information is obtained based on the second network model.

[0174] In the embodiment of the present disclosure, the first network model to be accelerated is obtained, and based on the first network model, first model information of the first network model is obtained;

[0175] Here, the first model information also includes: the input dimension and output dimension of the first network model, the number of layers of each network layer constituting the first network model, and the input and output relationship between the each network layer.

[0176] The first network model is pruned to obtain a pruned second network model, and second model information of the second network model is obtained based on the pruned second network model.

[0177] It should be noted that pruning technology reduces the complexity of the neural network model by removing redundant connections within the neural network model, thereby compressing the neural network model.

[0178] The embodiments of the present disclosure do not limit the specific method of pruning. For example, the first network model can be sparsely trained to obtain a first network model that has undergone sparse training, and some channels in the first network model that has undergone sparse training can be pruned to obtain a pruned second network model.

[0179] As an example, the first network model is taken as input, and unimportant channels are automatically identified during the training process, and the channels are simply pruned, so as to obtain a second network model with considerable accuracy and compactness; the levels of pruning may include weight dimension, convolution kernel dimension, channel dimension or layer dimension; it should be noted that sparse training in the weight dimension can obtain a high compression rate, but requires special software / hardware for acceleration, and sparse training in the layer dimension does not require special hardware, but has flexibility; sparse training in the channel dimension achieves a good balance between the flexibility of pruning and the difficulty of implementation, and can not only prune the convolutional layer, but also the fully connected layer.

[0180] The following provides a specific example in combination with any of the above technical solutions. The present disclosure provides a method for accelerating reasoning, such as Figure 3 As shown, Figure 3is a flowchart of a method for accelerating reasoning according to an exemplary embodiment. Figure 2 The method comprises:

[0181] Step S201, obtaining a first network model to be accelerated and the first model information of the first network model; performing pruning processing on the first network model to obtain the second network model, and obtaining the second model information based on the second network model;

[0182] In this example, the first network model is a pre-trained network model;

[0183] The first network model to be accelerated is accelerated in two ways. One way is to directly perform compilation and collaborative optimization on the first network model. The other way is to first prune the first network model to obtain a second network model, and then perform compilation and collaborative optimization on the second network model.

[0184] It should be noted that, for different first network models, the performance degradation of the second network model after pruning may be different. The second network model with larger performance degradation may not be suitable for the application environment corresponding to the first network model. However, this example is to build a general compilation collaborative optimization algorithm framework, so the two-way optimization method is more universal.

[0185] Step S202, inputting the first model information and the second model information into a fixed-point optimization module of an inference acceleration system; wherein the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel;

[0186] The hardware unit targeted by this example is DSP. In order to better utilize the advantages of DSP, the first network model and the second network model can be processed in a fixed-point manner;

[0187] By inputting the first model information and the second model information into the fixed-point optimization module of the inference acceleration system, the fixed-point bit width is optimized by using the first fixed-point optimization submodule and the second fixed-point optimization submodule set in parallel in the fixed-point optimization module, so as to obtain a fixed-point solution with better fixed-point effect.

[0188] like Figure 4 As shown, Figure 4 This is a schematic diagram of inference acceleration processing of a network model according to an exemplary embodiment. The first network model to be accelerated and the second network model after pruning can be coarsely fixed-point processed, and the first network model and the second network model after coarse fixed-point processing are input into the fixed-point optimization module of the inference acceleration system, and the first network model and the second network model are optimized for fixed-point bit width using the fixed-point optimization module.

[0189] Step S203, using the first fixed-point optimization submodule, based on the first model information and the second model information, randomly determining a first fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0190] In this example, the first model information includes the network layer type and network parameters of each network layer constituting the first network model; the second model information includes the network layer type and network parameters of each network layer constituting the second network model.

[0191] The first fixed-point optimization submodule may include: a random fixed-point bit width generation submodule; using the random fixed-point bit width generation submodule, for each network layer in the network model input into the first fixed-point optimization submodule, randomly determining the first fixed-point bit width of each network layer.

[0192] In this example, the random fixed-point bit width generation submodule randomly determines the first fixed-point bit width of each network layer in the first network model according to the input first model information; and randomly determines the second fixed-point bit width of each network layer in the second network model according to the input second model information.

[0193] Step S204: using the second fixed-point optimization submodule, performing sensitivity processing on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, determining the second fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0194] In this example, the second fixed-point optimization submodule may include: a bit width sensitivity analysis submodule, a hybrid random fixed-point bit width generation submodule;

[0195] The bit width sensitivity analysis submodule is used to perform bit width sensitivity analysis on the network parameters in the input network model to determine the bit width sensitivity corresponding to the network parameters of each network layer in the network model; the mixed random fixed-point bit width generation submodule is used to randomly determine the second fixed-point bit width of each network layer based on the bit width sensitivity corresponding to the network parameters of each network layer in the network model.

[0196] It should be noted that by performing a bit width sensitivity analysis on the network parameters of each network layer in the network model, the second fixed-point bit width of each network layer is determined according to the bit width sensitivity of the network parameters of each network layer. It can be understood that a network parameter with a smaller bit width sensitivity can be set to a smaller fixed-point bit width; a network parameter with a larger bit width sensitivity can be set to a larger fixed-point bit width.

[0197] Step S205, performing first fixed-point processing on network parameters in the first network model and the second network model according to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model;

[0198] In this example, the first fixed-point optimization submodule further includes: a fixed-point processing submodule; the second fixed-point optimization submodule further includes: a fixed-point processing submodule.

[0199] Using the fixed-point processing submodule in the first fixed-point optimization submodule, based on the first fixed-point bit width of the input network model, the network parameters of the network model are fixed-point processed to obtain the first fixed-point parameters of the network model;

[0200] The fixed-point processing submodule in the second fixed-point optimization submodule is used to perform fixed-point processing on the network parameters of the network model based on the second fixed-point bit width of the input network model to obtain second fixed-point parameters of the network model.

[0201] Step S206, determining a third network model and fixed-point configuration information corresponding to the third network model based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model;

[0202] In this example, the fixed-point optimization module further includes: a first discrimination submodule;

[0203] Using the first discrimination submodule, based on the first fixed-point parameter output by the first fixed-point optimization submodule and the second fixed-point parameter output by the second fixed-point optimization submodule, the processing accuracy of the first fixed-point parameter and the second fixed-point parameter is determined, and a target fixed-point parameter with a better fixed-point effect (i.e., better processing accuracy) is determined. Based on the target fixed-point parameter, the fixed-point configuration information of the accelerated network model is determined.

[0204] In some embodiments, the fixed-point configuration information corresponding to the third network model can be fed back to the first fixed-point optimization sub-module and the second fixed-point optimization sub-module; the fixed-point configuration information is used by the first fixed-point optimization sub-module and the second fixed-point optimization sub-module to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

[0205] In this example, the fixed-point configuration information output by the first discrimination submodule is used as input feedback information of the first fixed-point optimization submodule and the second fixed-point optimization submodule to guide the optimization process of the next fixed-point optimization module.

[0206] like Figure 5 As shown, Figure 5 The figure is a schematic diagram of a fixed-point optimization module according to an exemplary embodiment. The fixed-point optimization module is also optimized in two ways (i.e., the first fixed-point optimization submodule and the second fixed-point optimization submodule).

[0207] Step S207, inputting the third network model into the SIMD optimization module of the inference acceleration system, wherein the SIMD optimization module includes: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel;

[0208] In this example, after obtaining the fixed-point configuration information, SIMD optimization needs to be performed on the DSP hardware processing unit.

[0209] like Figure 4 As shown, the third network model output by the fixed-point optimization module can be input into the SIMD optimization module, and the third network model can be optimized in parallel using the first SIMD optimization submodule and the second SIMD optimization submodule.

[0210] Step S208, using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model; using the second SIMD optimization submodule, randomly extracting multiple operation instructions to be converted from the third network model, converting the multiple operation instructions into multiple SIMD instructions respectively, and obtaining a converted fifth network model;

[0211] In this example, the first SIMD optimization submodule includes: a random single SIMD optimization generation submodule; the second SIMD optimization submodule includes: a random mixed SIMD optimization generation submodule.

[0212] A random single SIMD optimization generation submodule is used to randomly extract an operation instruction from the input network model, and the operation instruction is converted into a SIMD instruction to obtain an optimized network model;

[0213] The random mixed SIMD optimization generation submodule is used to randomly extract multiple operation instructions from the input network model, and the operation instructions are converted into SIMD instructions to obtain an optimized network model.

[0214] Step S209, respectively determining the performance parameters and running time of the fourth network model and the fifth network model; based on the performance parameters and running time of the fourth network model and the fifth network model, determining the accelerated target network model and SIMD configuration information of the target network model;

[0215] In this example, the first SIMD optimization submodule includes: a performance test submodule and a time consumption test submodule; the second SIMD optimization submodule includes: a performance test submodule and a time consumption test submodule.

[0216] Determine the performance parameters and running time of the network model output by the random single SIMD optimization generation submodule by using the performance test submodule and the time consumption test submodule in the first SIMD optimization submodule;

[0217] The performance parameters and running time of the network model output by the random hybrid SIMD optimization generation submodule are determined by using the performance test submodule and the time consumption test submodule in the second SIMD optimization submodule.

[0218] The SIMD optimization module may include: a second determination submodule:

[0219] The second determination submodule is used to determine the accelerated target network model and the SIMD configuration information corresponding to the target network model based on the performance parameters and running time of the network model output by the first SIMD optimization submodule and the performance parameters and running time of the network model output by the second SIMD optimization submodule.

[0220] In some embodiments, the SIMD configuration information of the target network model may be fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

[0221] In this example, the SIMD configuration information output by the second determination submodule is used as input feedback information of the first SIMD optimization submodule and the second SIMD optimization submodule to guide the optimization process of the next SIMD optimization module.

[0222] like Figure 6 As shown, Figure 6 It is a structural diagram of a SIMD optimization module according to an exemplary embodiment. The SIMD optimization module is divided into two paths for parallel optimization; one path is a random single SIMD generation submodule to generate a series of single SIMD optimization solutions, and the other path is a random mixed SIMD generation submodule to generate a series of mixed SIMD optimization solutions. The SIMD optimization solutions generated by the two paths enter the performance test submodule and the time consumption test submodule respectively, and the second judgment submodule determines the better SIMD optimization solution.

[0223] Step S210: determining target model configuration information of the target network model according to the fixed-point configuration information and the SIMD configuration information.

[0224] In this example, after obtaining the fixed-point configuration information and the SIMD configuration information, the target model configuration information of the accelerated target network model is determined; the target model configuration information can be sent to the mobile edge device, so that the mobile edge device can build the accelerated target network model based on the target model configuration information and use the target network model for information processing.

[0225] The disclosed embodiment also provides an inference acceleration device. Figure 7 FIG. 1 is a schematic diagram showing the structure of an inference acceleration device according to an exemplary embodiment. Figure 7 As shown, the device 100 comprises:

[0226] An acquisition module 101 is used to acquire first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model;

[0227] The first optimization module 102 is used to input the first model information and the second model information into the inference acceleration system, and use the fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information; determine the third network model and the fixed-point configuration information of the third network model based on the first network model and the second network model after the fixed-point optimization processing;

[0228] The second optimization module 103 is used to optimize and accelerate the third network model based on SIMD instructions by using the single instruction multiple data (SIMD) optimization module of the inference acceleration system to obtain SIMD configuration information of the accelerated target network model;

[0229] The determination module 104 is used to determine the target model configuration information of the target network model according to the fixed-point configuration information and the SIMD configuration information.

[0230] Optionally, the fixed-point optimization module includes: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel;

[0231] The first optimization module 102 is used to:

[0232] Using the first fixed-point optimization submodule, based on the first model information and the second model information, randomly determine a first fixed-point bit width corresponding to each layer in the first network model and the second network model;

[0233] Utilizing the second fixed-point optimization submodule, sensitivity processing is performed on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, the second fixed-point bit width corresponding to each layer in the first network model and the second network model is determined.

[0234] Optionally, the first optimization module 102 is used to:

[0235] According to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, the network parameters in the first network model and the second network model are subjected to first fixed-point processing to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model;

[0236] Based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model, a third network model and fixed-point configuration information corresponding to the third network model are determined.

[0237] Optionally, the first optimization module 102 is used to:

[0238] The fixed-point configuration information corresponding to the third network model is fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule; the fixed-point configuration information is used by the first fixed-point optimization submodule and the second fixed-point optimization submodule to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

[0239] Optionally, the SIMD optimization module comprises: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel;

[0240] The second optimization module 103 is used to:

[0241] Using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model;

[0242] Utilizing the second SIMD optimization submodule, randomly extracting a plurality of operation instructions to be converted from the third network model, converting the plurality of operation instructions into a plurality of SIMD instructions respectively, to obtain a converted fifth network model; and determining the performance parameters and running time of the fourth network model and the fifth network model respectively;

[0243] Based on the performance parameters and running time of the fourth network model and the fifth network model, the target network model and SIMD configuration information of the target network model are determined.

[0244] Optionally, the second optimization module 103 is used to:

[0245] The SIMD configuration information of the target network model is fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

[0246] Optionally, the acquisition module 101 is used to:

[0247] Acquire a first network model to be accelerated and the first model information of the first network model;

[0248] The first network model is pruned to obtain the second network model, and the second model information is obtained based on the second network model.

[0249] Figure 8 800 is a block diagram of an inference acceleration device according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a mobile computer, etc.

[0250] Reference Figure 8 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0251] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0252] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0253] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0254] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0255] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0256] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0257] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0258] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0259] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0260] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0261] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0262] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for accelerating reasoning, characterized in that: The method comprises: Acquire first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model; Inputting the first model information and the second model information into an inference acceleration system, and using a fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model respectively based on the first model information and the second model information and the fixed-point bit width hardware resource amount of the computing unit; Based on the first network model and the second network model after fixed-point optimization processing, determine the fixed-point parameters of the first network model and the second network model; according to the fixed-point parameters of the first network model and the second network model, determine the third network model and the fixed-point configuration information of the third network model; the third network model is a network model in which the fixed-point parameters of the first network model and the second network model meet the preset accuracy conditions; using the single instruction multiple data stream SIMD optimization module of the inference acceleration system, convert at least one operation instruction in the third network model into a SIMD instruction, optimize and accelerate the third network model based on the SIMD instruction, and obtain the SIMD configuration information of the accelerated target network model; The target model configuration information of the target network model is determined according to the fixed-point configuration information and the SIMD configuration information.

2. The method according to claim 1, characterized in that The fixed-point optimization module comprises: a first fixed-point optimization submodule and a second fixed-point optimization submodule arranged in parallel; The fixed-point optimization module of the inference acceleration system is used to perform fixed-point optimization processing on the first network model and the second network model based on the first model information and the second model information and the fixed-point width hardware resource amount of the computing unit, including: Using the first fixed-point optimization submodule, based on the first model information, the second model information, and the amount of fixed-point bit width hardware resources, randomly determine a first fixed-point bit width corresponding to each layer in the first network model and the second network model; The second fixed-point optimization submodule is used to perform sensitivity processing on the first model information and the second model information to obtain the bit width sensitivity of the first model information and the second model information; based on the bit width sensitivity of the first model information and the second model information, the second fixed-point bit width corresponding to each layer in the first network model and the second network model is determined.

3. The method according to claim 2, characterized in that Determining fixed-point parameters of the first network model and the second network model based on the first network model and the second network model after fixed-point optimization processing; Determining a third network model and fixed-point configuration information of the third network model according to the fixed-point parameters of the first network model and the second network model includes: According to the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model, respectively, the network parameters in the first network model and the second network model are subjected to first fixed-point processing to obtain first fixed-point parameters and second fixed-point parameters of the first network model and the second network model; Based on the processing accuracy corresponding to the first fixed-point parameter and the second fixed-point parameter of the first network model and the second network model, a third network model and fixed-point configuration information corresponding to the third network model are determined.

4. The method according to claim 3, characterized in that: The method further comprises: The fixed-point configuration information corresponding to the third network model is fed back to the first fixed-point optimization submodule and the second fixed-point optimization submodule; the fixed-point configuration information is used by the first fixed-point optimization submodule and the second fixed-point optimization submodule to determine the first fixed-point bit width and the second fixed-point bit width corresponding to the first network model and the second network model.

5. The method according to claim 1, characterized in that The SIMD optimization module comprises: a first SIMD optimization submodule and a second SIMD optimization submodule arranged in parallel; The single instruction multiple data (SIMD) optimization module of the inference acceleration system is used to convert at least one operation instruction in the third network model into a SIMD instruction, and the third network model is optimized and accelerated based on the SIMD instruction to obtain SIMD configuration information of the accelerated target network model, including: Using the first SIMD optimization submodule, randomly extracting an operation instruction to be converted from the third network model, converting the operation instruction into a SIMD instruction, and obtaining a converted fourth network model; Utilizing the second SIMD optimization submodule, randomly extracting a plurality of operation instructions to be converted from the third network model, and converting the plurality of operation instructions into a plurality of SIMD instructions respectively, to obtain a converted fifth network model; Determining the performance parameters and running time of the fourth network model and the fifth network model respectively; Based on the performance parameters and running time of the fourth network model and the fifth network model, the target network model and SIMD configuration information of the target network model are determined.

6. The method according to claim 5, characterized in that The method comprises: The SIMD configuration information of the target network model is fed back to the first SIMD optimization submodule and the second SIMD optimization submodule, and the SIMD configuration information is used by the first SIMD optimization submodule and the second SIMD optimization submodule to determine the operation instructions to be converted into the SIMD instructions.

7. The method according to claim 1, characterized in that The obtaining first model information of the first network model to be accelerated and second model information of the second network model corresponding to the first network model includes: Acquire a first network model to be accelerated and the first model information of the first network model; The first network model is pruned to obtain the second network model, and the second model information is obtained based on the second network model.

8. An inference acceleration device, characterized in that: The device comprises: An acquisition module, used to acquire first model information of a first network model to be accelerated and second model information of a second network model corresponding to the first network model; wherein the second network model is a compressed model of the first network model; A first optimization module is used to input the first model information and the second model information into an inference acceleration system, and use the fixed-point optimization module of the inference acceleration system to perform fixed-point optimization processing on the first network model and the second network model respectively based on the first model information and the second model information and the fixed-point width hardware resource amount of the computing unit; based on the first network model and the second network model after the fixed-point optimization processing, determine the fixed-point parameters of the first network model and the second network model; according to the fixed-point parameters of the first network model and the second network model, determine the third network model and the fixed-point configuration information of the third network model; the third network model is a network model whose fixed-point parameters in the first network model and the second network model meet the preset accuracy conditions; A second optimization module is used to use the single instruction multiple data (SIMD) optimization module of the inference acceleration system to convert at least one operation instruction in the third network model into a SIMD instruction, optimize and accelerate the third network model based on the SIMD instruction, and obtain SIMD configuration information of the accelerated target network model; A determination module is used to determine the target model configuration information of the target network model according to the fixed-point configuration information and the SIMD configuration information.

9. An inference acceleration device, characterized in that: include: processor; a memory for storing executable instructions; The processor is configured to implement the inference acceleration method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an inference acceleration device, enables the inference acceleration device to execute the inference acceleration method described in any one of claims 1 to 7.