Deep learning model explanation method, apparatus, and device

Abstract

The embodiment of the application provides a deep learning model explanation method, device and equipment, the method comprises the following steps: loading a target deep learning model according to a loading mode corresponding to the target deep learning model, obtaining a model file of the target deep learning model; performing convolution, uniformization, pooling and activation processing on the model file of the target deep learning model, obtaining an analysis file corresponding to the model file; reconstructing the analysis file, obtaining a reconstructed file corresponding to the analysis file; and representing the reconstructed file as a structure description file containing only structure description and a weight file arranged according to the order of operators in the structure description file. The method provided by the application can express models under different training frameworks as unified network structure representation and unified parameter arrangement order, improving the efficiency of deep learning model deployment.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of deep learning technology, specifically to a method, apparatus, and device for interpreting deep learning models. Background Technology

[0002] We are currently in an era of rapid rise in Artificial Intelligence (AI) technology, with major technology companies launching their own deep learning training frameworks, such as Google's TensorFlow, Facebook's PyTorch, and Amazon's MXNet. A large number of deep learning professionals are using these tools to develop models, designing many excellent AI models and propelling the rapid development of the artificial intelligence industry.

[0003] As AI technology continues to advance, its application scenarios are becoming increasingly broad. AI deployment is a crucial step in realizing AI functionality. However, building resource-intensive deep learning frameworks at the edge does not meet practical needs and would result in significant resource waste. Therefore, model parsing is an indispensable part of the AI ​​edge deployment process. However, due to various reasons such as the different storage structures of AI models trained under different AI frameworks, the different operator expressions under different AI frameworks, and the different operator calculation methods on different hardware platforms, the difficulty of deploying deep learning models is increased, leading to low deployment efficiency. Summary of the Invention

[0004] This invention provides a method, apparatus, and device for interpreting deep learning models, which solves the problem of low deployment efficiency of deep learning models in existing methods.

[0005] In a first aspect, embodiments of the present invention provide a method for interpreting deep learning models, including:

[0006] Load the target deep learning model according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model;

[0007] The model file of the target deep learning model is processed by convolution, normalization, pooling and activation to obtain the parsed file corresponding to the model file;

[0008] The parsed file is reconstructed to obtain the corresponding reconstructed file.

[0009] The reconstructed file is represented as a structure description file containing only the structure description and a weight file arranged in the order of the operators in the structure description file.

[0010] In one embodiment, the target deep learning model is loaded according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model, including:

[0011] Load the target deep learning model to obtain a model execution computation graph file containing the structure and parameters. The model execution computation graph file includes a node array, an input array, an output array, and an initial value array. The node array is used to store the computation nodes of the model, the input array is used to store the input nodes of the model, the output array is used to store the output nodes of the model, and the initial value array is used to store the weight parameters of the model.

[0012] In one embodiment, the method further includes:

[0013] Extract the input size description of the model's input nodes from the input array;

[0014] Extract the names and shapes of the model's weight parameters from the initial value array;

[0015] Extract the network operator names and structural descriptions of the computation nodes of the model from the node array.

[0016] In one embodiment, the parsed file is reconstructed to obtain a reconstructed file corresponding to the parsed file, including:

[0017] The weight parameters, type descriptions, and parameter descriptions of the operators in the forward inference computation are reconstructed according to the operator names.

[0018] In one embodiment, the method further includes converting floating-point parameters into fixed-point weight parameters during the reconstruction process.

[0019] In one embodiment, converting floating-point parameters to fixed-point weight parameters includes:

[0020] Convert floating-point parameters to fixed-point weighted parameters using the following expression:

[0021]

[0022] Among them, w float represents a floating-point parameter, and w represents a fixed-point weight parameter.

[0023] In one embodiment, the target deep learning model includes an ONNX model.

[0024] In a second aspect, embodiments of the present invention provide a deep learning model interpretation apparatus, comprising:

[0025] The loading module is used to load the target deep learning model according to the loading method corresponding to the target deep learning model, and obtain the model file of the target deep learning model;

[0026] The parsing module is used to perform convolution, normalization, pooling, and activation processing on the model file of the target deep learning model to obtain the parsing file corresponding to the model file;

[0027] The reconstruction module is used to reconstruct the parsed file to obtain the reconstructed file corresponding to the parsed file;

[0028] The interpretation module is used to represent the reconstructed file as a structure description file containing only the structure description and a weight file arranged in the order of the operators in the structure description file.

[0029] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0030] At least one processor and memory;

[0031] The memory stores instructions that the computer executes;

[0032] At least one processor executes computer execution instructions stored in memory, causing the at least one processor to perform the deep learning model interpretation method as described in any of the first aspects.

[0033] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the deep learning model interpretation method as described in any of the first aspects.

[0034] The deep learning model interpretation method, apparatus, and device provided in this invention load a target deep learning model according to the loading method corresponding to the target deep learning model to obtain a model file of the target deep learning model; perform convolution, normalization, pooling, and activation processing on the model file of the target deep learning model to obtain a parsing file corresponding to the model file; reconstruct the parsing file to obtain a reconstructed file corresponding to the parsing file; and represent the reconstructed file as a structure description file containing only the structure description and a weight file arranged according to the operator order in the structure description file. This enables models under different training frameworks to be expressed with a unified network structure representation and a unified parameter arrangement order, improving the efficiency of deep learning model deployment. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0036] Figure 1 A flowchart of a deep learning model interpretation method provided in an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the structure of a deep learning model interpretation device provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0039] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0041] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0042] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0043] Major technology companies have launched their own deep learning training frameworks, such as Google's TensorFlow, Facebook's PyTorch, and Amazon's MXNet. While numerous deep learning professionals have used these tools to develop models and design many excellent AI models, propelling the rapid development of the artificial intelligence industry, the deployment of deep learning models has become increasingly difficult due to various factors, including differences in storage structures, operator representations, and operator computation methods across different hardware platforms. To address the computational characteristics of AI models, different manufacturers have produced various hardware acceleration units, such as GPUs, TPUs, and NPUs. These different types of hardware acceleration units from different manufacturers implement different computational modes and operator implementations. This application proposes a deep learning model interpretation method for a self-developed KDP acceleration unit, capable of parsing multiple deep learning models and optimizing the AI ​​deployment workflow. By analyzing AI models under different training frameworks and representing them according to a unified network structure and following a unified parameter arrangement order, the process of analyzing and debugging different models during AI deployment is saved for practitioners. This facilitates subsequent optimization of the unified operator and improves the efficiency of AI model deployment. The method of this application will be described in detail below through specific embodiments.

[0044] Figure 1 This is a flowchart illustrating a deep learning model interpretation method provided in an embodiment of the present invention. Figure 1 As shown, the deep learning model interpretation method provided in this embodiment may include:

[0045] S101. Load the target deep learning model according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model.

[0046] Understandably, deep learning models obtained using different deep learning training frameworks have different model files, and different model files require different loading methods. The model structures obtained under different deep learning frameworks differ, requiring loading and parsing using the loading functions corresponding to the framework environment of the model format. For example, model.onnx uses the loading function in the ONNX environment, and model.h5 uses the loading function in the Keras environment. Therefore, a correspondence between deep learning models and loading methods (a correspondence between model files and loading methods) can be established in advance. Based on this pre-established correspondence, the target deep learning model can be loaded according to the loading method corresponding to the target deep learning model, thus obtaining the model file of the target deep learning model. Specifically, different deep learning model framework environments can be set up, and the appropriate model loading method can be selected according to different model files to import the deep learning model of the corresponding framework for subsequent model parsing.

[0047] S102. Perform convolution, normalization, pooling, and activation processing on the model file of the target deep learning model to obtain the parsed file corresponding to the model file.

[0048] After obtaining the model file of the target deep learning model, it needs to be parsed. For example, Python can be used to parse it, expressing it as a unified network structure representation and a unified parameter arrangement order, thus saving engineers the process of parsing and debugging different models when deploying AI chips. Specifically, parsing the model file of the target deep learning model can include: convolution processing (CONV), normalization processing (BatchNorm), pooling processing (POOL), and activation processing (activation).

[0049] After loading models under different frameworks, an editable visual structure file for that framework is obtained. This file contains a lot of information and configurations unrelated to inference. When deploying the model on the self-developed hardware platform, only the sequence of operators, along with their corresponding operator descriptions and parameters, are needed. Therefore, during parsing, the operators can be extracted from the model and the corresponding transformations can be performed.

[0050] S103. Reconstruct the parsed file to obtain the reconstructed file corresponding to the parsed file.

[0051] After obtaining the parsed file corresponding to the model file, it needs to be rewritten. Specifically, this can involve applying new convolution processing (New_CONV), new uniformization processing (New_BatchNorm), new pooling processing (New_POOL), and new activation processing (New_activation). After rewriting, the rewritten file corresponding to the parsed file can be obtained.

[0052] S104. Represent the reconstructed file as a structure description file containing only the structure description and a weight file arranged in the order of the operators in the structure description file.

[0053] Finally, following a unified network structure and a unified parameter arrangement order, the reconstructed file obtained after the transformation is represented as a structure description file (structure.cfg) containing only the structure description and a weight file (weight.wt) arranged according to the operator order in the structure description file.

[0054] The deep learning model interpretation method provided in this embodiment loads the target deep learning model according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model; performs convolution, normalization, pooling and activation processing on the model file of the target deep learning model to obtain the parsing file corresponding to the model file; reconstructs the parsing file to obtain the reconstructed file corresponding to the parsing file; and represents the reconstructed file as a structure description file containing only the structure description and a weight file arranged according to the operator order in the structure description file. It can express models under different training frameworks as a unified network structure representation and a unified parameter arrangement order, thereby improving the efficiency of deep learning model deployment.

[0055] Different frameworks load different model structures, have different operator descriptions, and have different operator parameter storage methods. Therefore, it is necessary to extract and convert the corresponding formats according to different models. Based on the above embodiments, the deep learning model interpretation method provided in this application will be further elaborated below using the ONNX model as an example.

[0056] Based on the above embodiments, the deep learning model interpretation method provided in this embodiment loads the target deep learning model according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model. Specifically, it may include: loading the target deep learning model to obtain a model execution operation graph file containing structure and parameters. The model execution operation graph file includes a node array, an input array, an output array, and an initial value array. The node array is used to store the computation nodes of the model, the input array is used to store the input nodes of the model, the output array is used to store the output nodes of the model, and the initial value array is used to store the weight parameters of the model.

[0057] For example, after loading an ONNX model, you get a ModelProto file. Ignore the parts of this file that are irrelevant to forward inference, and only parse the GraphProto (the computation graph executed by the model) containing the structure and parameters. The GraphProto contains four arrays: node, input, output, and initializer. The node array stores all the computation nodes in the model, the input array stores all the input nodes, the output array stores all the output nodes, and the initializer stores all the weight parameters of the model.

[0058] Understandably, given the vast amount of information contained in the model, much of which is useless during deployment, it's necessary to extract the operator descriptions required for deployment on the self-developed hardware platform. This necessitates extracting information from four different arrays. Specifically, this involves extracting the input size descriptions of the model's input nodes from the input array; extracting the names and shapes of the model's weight parameters from the initialization array; and extracting the network operator names and structural descriptions of the model's computation nodes from the node array. For example, extracting the input size from the input array to reconstruct a new input description; extracting the names, shapes, and parameters from the initializer; and extracting the network operator names and structural descriptions from the node. It's important to note that the output parameter only exists within the ONNX model; it is discarded during parsing because it has no effect on model deployment.

[0059] Each operator in the ONNX model has a unique name. Using this name as a reference, each operator is rewritten and labeled, and redundant descriptions in unnecessary inference calculations are removed, retaining only the parts needed for deployment inference, and providing a unified description. Taking a self-developed AI chip as an example, based on its computational characteristics, only the weight parameters in the operator's forward inference calculation, as well as the operator's type and parameter descriptions, are retained. In other words, based on the above embodiments, the deep learning model interpretation method provided in this embodiment reconstructs the parsing file to obtain a reconstructed file corresponding to the parsing file. Specifically, this may include reconstructing the weight parameters in the operator's forward inference calculation, as well as the operator's type and parameter descriptions, according to the operator's name.

[0060] Based on the computational characteristics of the self-developed AI chip, for operators with multiple inputs, computation needs to be performed according to the computation order of the input operators. A route operator description is added, with the description being a negative number -n, indicating that the output of the current operator is taken as one of the inputs after counting n operators back to the current operator. Since the KDP hardware computation acceleration module's acceleration unit only supports fixed-point computation with 10-bit weight parameters for convolution operators, the floating-point parameter w needs to be converted during the reconstruction process. float Convert to fixed-point weight parameters w.

[0061] Specifically, converting floating-point parameters to fixed-point weighted parameters is done using the following expression:

[0062]

[0063] Among them, w float represents a floating-point parameter, and w represents a fixed-point weight parameter.

[0064] Since the standard homogenization operator includes four parameters—mu(μ), sigma(σ), beta(β), and gamma(γ)—while the KDP acceleration unit only requires two parameters, new_gamma(γ') and new_beta(β'), the four parameters are converted into the required two parameters, new_gamma(γ') and new_beta(β'), according to the following expression:

[0065]

[0066]

[0067] bias, new_gamma, and new_beta are all calculated in 16-bit format. The expression for the fixed-point conversion is:

[0068]

[0069] f(x) is the 16-bit fixed-point value after conversion, and x is the floating-point value before conversion.

[0070] The transformed parameters are then stored in the order of bias, new_beta, new_gamma, and weight per channel. Finally, the transformed structure and parameters are uniformly stored as a structure description file containing only the structure description and a weight file arranged according to the operator order in the structure description file.

[0071] In summary, the deep learning model interpretation method provided in this application can express deep learning models under different training frameworks into a unified network structure representation and a unified parameter arrangement order through parsing based on the computational characteristics of the operators. This can save engineers the parsing and debugging process of different models during chip deployment, facilitate subsequent optimization of the unified operators, and improve the deployment efficiency of deep learning models.

[0072] Figure 2 This is a schematic diagram of the structure of a deep learning model interpretation device provided in an embodiment of the present invention. Figure 2 As shown, the deep learning model interpretation device 20 provided in this embodiment may include: a loading module 201, a parsing module 202, a reconstruction module 203, and an interpretation module 204.

[0073] Loading module 201 is used to load the target deep learning model according to the loading method corresponding to the target deep learning model, and obtain the model file of the target deep learning model;

[0074] The parsing module 202 is used to perform convolution, normalization, pooling and activation processing on the model file of the target deep learning model to obtain the parsing file corresponding to the model file;

[0075] Reconstruction module 203 is used to reconstruct the parsed file to obtain the reconstructed file corresponding to the parsed file;

[0076] Interpretation module 204 is used to represent the reconstructed file as a structure description file containing only the structure description and a weight file arranged in the order of the operators in the structure description file.

[0077] The device provided in this embodiment can be used to perform... Figure 1 The technical solutions of the corresponding method embodiments are similar in principle and technical effect, and will not be described again here.

[0078] In one optional implementation, the loading module 201 is used to load the target deep learning model according to the loading method corresponding to the target deep learning model, thereby obtaining the model file of the target deep learning model, which may specifically include:

[0079] Load the target deep learning model to obtain a model execution computation graph file containing the structure and parameters. The model execution computation graph file includes a node array, an input array, an output array, and an initial value array. The node array is used to store the computation nodes of the model, the input array is used to store the input nodes of the model, the output array is used to store the output nodes of the model, and the initial value array is used to store the weight parameters of the model.

[0080] In an optional implementation, the loading module 201 is further configured to:

[0081] Extract the input size description of the model's input nodes from the input array;

[0082] Extract the names and shapes of the model's weight parameters from the initial value array;

[0083] Extract the network operator names and structural descriptions of the computation nodes of the model from the node array.

[0084] In one optional implementation, the reconstruction module 203 is used to reconstruct the parsed file to obtain a reconstructed file corresponding to the parsed file, which may specifically include:

[0085] The weight parameters, type descriptions, and parameter descriptions of the operators in the forward inference computation are reconstructed according to the operator names.

[0086] In one optional implementation, the reconstruction module 203 is further configured to convert floating-point parameters into fixed-point weight parameters during the reconstruction process.

[0087] In one optional implementation, converting floating-point parameters to fixed-point weight parameters includes:

[0088] Convert floating-point parameters to fixed-point weighted parameters using the following expression:

[0089]

[0090] Among them, w float represents a floating-point parameter, and w represents a fixed-point weight parameter.

[0091] In one alternative implementation, the target deep learning model includes an ONNX model.

[0092] This invention also provides an electronic device, please refer to [link to relevant documentation]. Figure 3 As shown, the embodiments of the present invention are only used as examples. Figure 3 The examples are provided for illustration only and do not imply that the invention is limited to these examples. Figure 3 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. Figure 3 As shown, the electronic device 30 provided in this embodiment may include: a memory 301, a processor 302, and a bus 303. The bus 303 is used to connect the various components.

[0093] The memory 301 stores a computer program, which, when executed by the processor 302, can implement the technical solutions of any of the above method embodiments.

[0094] The memory 301 and the processor 302 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines, such as bus 303. The memory 301 stores a computer program that implements a deep learning model interpretation method, including at least one software functional module that can be stored in the memory 301 in the form of software or firmware. The processor 302 executes various functional applications and data processing by running the software program and modules stored in the memory 301.

[0095] The memory 301 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 301 stores programs, and the processor 302 executes the programs after receiving execution instructions. Furthermore, the software programs and modules within the memory 301 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.) and can communicate with various hardware or software components to provide an operating environment for other software components.

[0096] Processor 302 can be an integrated circuit chip with signal processing capabilities. The aforementioned processor 302 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. It is understood that... Figure 3 The structure shown is for illustrative purposes only and may include more... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented in hardware and / or software.

[0097] This invention also provides a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the technical solutions of any of the above method embodiments.

[0098] The various embodiments in this disclosure are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0099] The scope of protection of this disclosure is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its scope and spirit. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A method for interpreting deep learning models, characterized in that, The method is applied to model deployment on a hardware platform, including: The target deep learning model is loaded according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model; The model file of the target deep learning model is subjected to convolution, normalization, pooling and activation processing to obtain the parsed file corresponding to the model file; The parsed file is reconstructed to obtain the reconstructed file corresponding to the parsed file; wherein, during the reconstruction process, floating-point parameters are converted into fixed-point weight parameters; Based on a unified network structure and a unified parameter arrangement order, the reconstructed file is represented as a structure description file containing only the structure description and a weight file arranged according to the operator order in the structure description file.

2. The method according to claim 1, characterized in that, The step of loading the target deep learning model according to the loading method corresponding to the target deep learning model to obtain the model file of the target deep learning model includes: Load the target deep learning model to obtain a model execution computation graph file containing structure and parameters. The model execution computation graph file includes a node array, an input array, an output array, and an initial value array. The node array is used to store the computation nodes of the model, the input array is used to store the input nodes of the model, the output array is used to store the output nodes of the model, and the initial value array is used to store the weight parameters of the model.

3. The method according to claim 2, characterized in that, The method further includes: Extract the input size description of the model's input nodes from the input array; Extract the names and shapes of the model's weight parameters from the initial value array; Extract the network operator names and structural descriptions of the computation nodes of the model from the node array.

4. The method according to claim 3, characterized in that, The process of reconstructing the parsed file to obtain a reconstructed file corresponding to the parsed file includes: The weight parameters, type descriptions, and parameter descriptions of the operators in the forward inference computation are reconstructed according to the operator names.

5. The method according to claim 1, characterized in that, The process of converting floating-point parameters to fixed-point weight parameters includes: Convert floating-point parameters to fixed-point weighted parameters using the following expression: in, Represents floating-point parameters. This represents the fixed-point weight parameter.

6. The method according to any one of claims 1-5, characterized in that, The target deep learning model includes the ONNX model.

7. A deep learning model interpretation device, characterized in that, The device is used for model deployment on a hardware platform, including: The loading module is used to load the target deep learning model according to the loading method corresponding to the target deep learning model, so as to obtain the model file of the target deep learning model; The parsing module is used to perform convolution, normalization, pooling, and activation processing on the model file of the target deep learning model to obtain the parsing file corresponding to the model file; The reconstruction module is used to reconstruct the parsed file to obtain the reconstructed file corresponding to the parsed file; wherein, during the reconstruction process, floating-point parameters are converted into fixed-point weight parameters; The interpretation module is used to represent the reconstructed file as a structure description file containing only the structure description and a weight file arranged according to the operator order in the structure description file, according to a unified network structure and a unified parameter arrangement order.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the deep learning model interpretation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the deep learning model interpretation method as described in any one of claims 1-6.

Images