Autonomous controllable chip model automatic deployment method and system based on multi-level intermediate representation
The method of automatic deployment of autonomous and controllable chip models through multi-level intermediate representation solves the problems of low model deployment efficiency and poor controllability on autonomous and controllable chip platforms, and realizes efficient, autonomous and universal model deployment on different chip platforms, which is suitable for unmanned cluster tasks.
Patent Information
- Application Number
- CN202211531031.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-01
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-12-01
AI Technical Summary
Existing technologies for deploying artificial intelligence models on independently controllable chip platforms suffer from low deployment efficiency, poor controllability, and insufficient versatility. In particular, in unmanned cluster tasks, it is difficult to rapidly iterate and adapt to diverse chip platforms in offline environments.
An automatic deployment method for autonomous and controllable chip models based on multi-level intermediate representation is adopted. Through a model parsing module, a model intermediate representation module, and a model deployment module, it can realize the rapid parsing, formatted storage, and automatic deployment of deep learning models, and support operator conversion and model optimization for different chip platforms.
It improves the efficiency and autonomy of model deployment, reduces development workload, enhances the adaptability and deployment performance of models on different chip platforms, supports the development of custom operators, and enhances the controllability and versatility of model deployment.
Smart Images

Figure CN116339752B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology and relates to an automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation. Background Technology
[0002] In recent years, with the development of artificial intelligence technology, deep neural networks have become the main technical approach to endow vehicle-mounted, airborne, spaceborne, and security equipment with intelligent processing capabilities. Especially in unmanned swarm reconnaissance missions, information analysis methods based on convolutional neural networks are of great significance for unmanned swarms to complete tasks such as target detection and situational analysis.
[0003] Given the characteristics of unmanned swarm reconnaissance missions, unmanned swarms need to simultaneously conduct battlefield situation reconnaissance, route planning, and intelligent coordination. The intelligent computing mission planning needs to comprehensively consider factors such as information security, data latency, network environment, and communication bandwidth.
[0004] Because each terminal of the unmanned swarm needs to perform independent intelligent computing tasks on the information it detects simultaneously, and communication between the unmanned swarm and the ground is limited during the task execution, the necessity of processing the raw information on the airborne terminal is highlighted. In unmanned swarm missions, the migration of the main intelligent computing power from the cloud to the edge or terminal has become an inevitable trend.
[0005] In terminal devices, general-purpose processors such as ARM processors are not well-suited to the computational demands of large-scale multiply-accumulate operations in deep neural networks, resulting in low computational efficiency. With the increasing demand for AI applications and the requirements for domestic production and independent control of hardware in specialized fields, domestically produced intelligent chips for deep neural networks have seen significant development. These chips, built upon traditional processors, incorporate dedicated AI hardware acceleration functions into terminal devices, leading to a variety of domestically produced chips with different performance characteristics. This has greatly improved the intelligent data analysis and processing capabilities of terminal computing devices while simultaneously ensuring information security and technological self-control.
[0006] The application of artificial intelligence algorithms on domestically developed and controllable chip platforms requires frequent iterative design and deployment of AI algorithm models based on the needs of different scenarios and tasks. For the same model, continuous training is needed based on newly added scenario data, updating model weights and deploying the model on the chip platform. During model deployment, due to the diversity of target chip deployment platforms, especially domestically developed and controllable chip platforms, and the specificity of their hardware and software platforms, the model deployment task currently faces the following problems:
[0007] 1) Existing model deployment methods for various training frameworks are highly coupled with publicly available model deployment tool software versions. The models to be deployed come from different types and software versions of artificial intelligence model training frameworks. In offline environments, the configuration process for deployment tool software environments is cumbersome and unsuitable for the rapid iteration requirements of unmanned swarm artificial intelligence application scenarios.
[0008] 2) The general deployment method has low deployment autonomy, limited openness of the underlying source code, uncontrollable deployment effect of the model, and low support for non-general artificial intelligence operators.
[0009] 3) Currently, there are many types of domestically developed and controllable chip platforms. Specialized deployment solutions need to be developed for different model training platforms and different chip platforms, which has a long development cycle and a large workload. Summary of the Invention
[0010] To address the shortcomings of existing AI model deployment methods, and considering the deployment requirements of AI algorithms in current unmanned swarm tasks, this paper proposes an automatic deployment method for autonomous and controllable chip models based on multi-level intermediate representations. This method solves many problems associated with deploying deep learning network models on autonomous and controllable chips, including:
[0011] 1) The problem of rapid iterative deployment of deep learning network models in offline environments on domestically controllable chip platforms.
[0012] 2) Existing methods for deploying artificial intelligence models have low deployment controllability.
[0013] 3) The issues of the universality and efficiency of AI model deployment technologies across different chip platforms.
[0014] The technical solution adopted in this invention is as follows:
[0015] An automated deployment system for an autonomous and controllable chip model based on multi-level intermediate representation, the automated deployment system comprising:
[0016] Model parsing module: used to load the deep learning model trained from the model training platform, and parse the model network structure, operation types and operation parameters of each layer of the deep learning model;
[0017] Model intermediate representation module: This module defines a general model intermediate representation method, which is used to format and store the deep learning model parsed by the model parsing module into a general intermediate format model.
[0018] Model deployment module: It is used to traverse the intermediate format model saved in the intermediate format, perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms, and complete the automatic deployment of deep learning models on different domestically controlled chips.
[0019] Furthermore, the model parsing module includes a model loading submodule and a model analysis submodule;
[0020] The model loading submodule is used to load model formats from different training platforms such as PyTorch, Tensorflow, and Caffe; deserialize the deep learning models of each training platform according to the model format of different platforms to obtain the operational characteristics of each node, node connection relationships and weight parameters of the model; convert the models of different formats into static graph models; and reserialize and save the model weight parameters.
[0021] The model analysis submodule is used to parse the static graph model; develop a static graph traversal method to obtain the connection relationships of each node in the static graph model, the correspondence between operator types and parameterized network nodes and weight parameters, based on the structure and node names of the static graph model; develop an operator size calculation library to calculate the size of each node's feature map based on the computational characteristics of different operators and the size of the input feature map; and develop an operator parsing library to obtain a complete set of parameters controlling the characteristics of each operation node from the static graph format data based on the characteristics of different operator operations, and reserve a parameter saving interface to facilitate the saving of characteristic parameter sets corresponding to different operator operations.
[0022] Furthermore, the model intermediate representation module includes a model middleware storage submodule and a model middleware loading and access submodule;
[0023] The model middleware storage submodule is an embeddable module. First, the operation-level parameter descriptions in the model middle representation module are aligned with the characteristic parameter sets corresponding to each operator operation in the model parsing module. Then, the model middleware storage submodule is connected to the operator parsing library of the model parsing module. Finally, the static graph nodes are traversed using the static graph traversal method of the model parsing module, and the corresponding operator parsing library is called according to the operator type mapping. The connected model middleware storage submodule is used to automatically generate a general model middleware.
[0024] The model middleware loading and access submodule is used to complete the full loading of the model middleware, and to reassign parameters, replace operations, and merge operations in the model.
[0025] The model middleware loading and access submodule automatically generates middleware model loading code based on the rules of the multi-level description of the model, by maintaining the correspondence table of operator parameter loading methods and the correspondence table of operator parameter data types, and designing code generation rules. When adding a new operator, it is only necessary to update the operation-level parameter description of the model middleware description, and the corresponding operator loading method can be automatically generated.
[0026] The model middleware loading and access submodule locates the operator nodes that need to be updated or deleted based on the network-level description of the network node number, node type, and node input-output connection relationship of the middleware model to be accessed. It can also freely access and manipulate the characteristic parameters of the corresponding operators, reconstruct the intermediate representation model as needed, and save the reconstructed intermediate format model.
[0027] Furthermore, the model deployment module includes a deployment platform interface library, a deployment platform operator conversion library, and a deployment platform hardware language supplementary library;
[0028] The deployment platform interface library is used for accessing and developing development tools at different chip levels, including a deployment engine initialization module, a middleware model loading module, a model traversal and conversion module, and a deployment engine generation module. The deployment engine initialization module and deployment engine generation module related to the deployment engine are encapsulated and developed according to the development interfaces of different chip platforms.
[0029] The deployment engine initialization module is coupled with the method for generating deployment engine files for different chip platforms, and is used for predefining data structures and initializing special operations required for generating deployment engine files for different chip platforms.
[0030] The middleware model loading module automatically generates the model based on the intermediate representation module, and completes the loading and free access to the parameters of the intermediate format model.
[0031] The model traversal and conversion module, based on the characteristics of different chips, calls the deployment platform operator conversion library to convert the operators of the intermediate format model into high-speed computing operators supported by the corresponding deployment platform. During the traversal process, it maintains the connection relationship of the operations of each node of the deployment engine, and completes the conversion of the entire model to the target deployment platform format. If there are operators in the model that are not supported by the target deployment platform, they are marked and serialized for storage, which facilitates the subsequent development of custom operators.
[0032] The deployment engine generation module generates a deployment engine for the corresponding chip platform based on the successfully traversed deployment engine stream. The deployment engine generation module can modify the input and output nodes of the deployment engine stream to generate deployment engines corresponding to different continuous parts of the model, thereby completing model splicing and model segmentation, and improving the freedom of model deployment.
[0033] Furthermore, the deployment platform operator conversion library is developed at the network model operator level, according to the different support conditions of different chip development tools, to create an operator conversion library corresponding to the chip platform. The deployment platform operator conversion library converts the general operators in the network-level parameter description, operation-level parameter description, and operator parameter characteristic-level description into high-speed computing operators supported by the deployment platform, which is achieved through operator type mapping and operator parameter conversion.
[0034] The operator type mapping is based on the operator support of the deployment platform, maintaining a one-to-one correspondence between general operators and high-speed computing operators of the deployment platform; through the mapping table, the operator operations described by the intermediate format model during model traversal can automatically correspond to the high-speed computing operator interfaces of the same operations supported by the chip platform.
[0035] The operator parameter conversion involves transforming the operator parameters of the intermediate format model according to the parameter interface characteristics of the high-speed computing operator on the chip platform, so that the parameters conform to the calling logic of the high-speed computing operator, and assigning values in the high-speed computing operator calling interface to complete the operator conversion.
[0036] Furthermore, the deployment platform hardware language supplementary library is a supplementary library developed at the deployment platform hardware language level, based on the different levels of chip openness, using the corresponding deployment platform hardware language to develop corresponding chip platforms; it is used to implement functions not officially supported by the chip platform, including custom operator development, model optimization, and efficient model pre- and post-processing development;
[0037] The development of custom operators involves analyzing the computational principles of unsupported operators based on the list of unsupported operators obtained from the model traversal module, writing corresponding computational operations using the hardware language of the chip platform, and then encapsulating the written operators into custom operators using chip development tools, adding them to the operator conversion library, and updating the operator mapping table.
[0038] Model optimization involves performing model quantization, operation merging, and operator optimization based on the hardware characteristics of different chips and their corresponding hardware languages, so that the deployed model can achieve better running performance on the corresponding domestically controlled chip.
[0039] High-efficiency model pre- and post-processing development involves developing efficient model pre- and post-processing operations using chip hardware languages based on the pre- and post-processing operation principles corresponding to different models. This enhances the model deployment capabilities of the corresponding chip deployment platform, reduces computational operations on non-chip devices, and improves deployment efficiency and performance.
[0040] An automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation, wherein the automatic deployment method employs the aforementioned automatic deployment system, and the automatic deployment method includes:
[0041] (1) Load the deep learning model trained from the model training platform through the model parsing module, and parse the model network structure, operation type and operation parameters of each layer of the deep learning model;
[0042] (2) A general model intermediate representation method is defined by using the model intermediate representation module. The deep learning model parsed in step (1) is formatted and stored as a general intermediate format model through the model intermediate representation method.
[0043] (3) Using the model deployment module, traverse the intermediate format model saved in the intermediate format in step (2), and perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms to complete the automatic deployment of deep learning models on different independent and controllable chips.
[0044] Furthermore, in step (2), the deep learning model parsed in step (1) is first described in a multi-level complete manner using the model intermediate representation method. Then, the model middleware storage submodule and the model middleware loading and access submodule are used to perform precise model conversion, model quantization, model pruning, and model operation merging.
[0045] Furthermore, the method of using the intermediate representation of the model to perform a multi-level complete description of the deep learning model parsed in step (1) includes the following specific methods:
[0046] Multi-level descriptions include network-level parameter descriptions, operation-level parameter descriptions, and operator parameter characteristic-level descriptions;
[0047] The network-level parameter description is the highest level of model description, including the size of the input feature map of the deep learning network, the connection relationship of each network node, the operator type of each network node, the storage sequence number of the network node, and the correspondence between weighted nodes and weight parameter files.
[0048] The operation-level parameter description is an intermediate layer in the model description, and is described separately for different operator operations;
[0049] The operator parameter characteristic level description is the lowest level of the model description, including the data type, precision, and whether the parameter is a sequence.
[0050] Furthermore, in step (3), the platform interface library, the platform operator conversion library, and the platform hardware language supplementary library are used to enable access and development of multiple independent and controllable chip deployment platforms from top to bottom and multi-level, and to complete the automatic deployment of the model on different chip platforms.
[0051] Compared with the prior art, the present invention has the following advantages:
[0052] 1) This invention proposes a highly versatile automatic deployment method for autonomous and controllable chip models based on multi-level intermediate representations. Compared with existing deployment methods, it can significantly reduce the workload of deploying models on different chips and improve deployment efficiency.
[0053] 2) The automatic deployment method of the autonomous and controllable chip model based on multi-level intermediate representation proposed in this invention is independently developed. Compared with the existing deployment methods, the deployment method provided by this invention has a higher degree of control and supports the deployment and development of custom operators, making it more autonomous and controllable in terms of technology.
[0054] 3) The automatic deployment method for autonomous and controllable chip models based on multi-level intermediate representation proposed in this invention has less software dependency, decouples the training platform software type and version of the deployed model, and makes the entire model deployment scheme more portable and universal. Attached Figure Description
[0055] Figure 1 This is a flowchart of the automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation in an embodiment of the present invention. Detailed Implementation
[0056] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0057] This embodiment provides an automatic deployment system for an autonomous and controllable chip model based on multi-level intermediate representation. The automatic deployment system includes:
[0058] Model parsing module: used to load the deep learning model trained from the model training platform, and parse the model network structure, operation types and operation parameters of each layer of the deep learning model;
[0059] Model intermediate representation module: This module defines a general model intermediate representation method, which is used to format and store the deep learning model parsed by the model parsing module into a general intermediate format model.
[0060] Model deployment module: It is used to traverse the intermediate format model saved in the intermediate format, perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms, and complete the automatic deployment of deep learning models on different domestically controlled chips.
[0061] In this embodiment, the model parsing module includes a model loading submodule and a model analysis submodule;
[0062] The model loading submodule is used to load model formats from different training platforms such as PyTorch, Tensorflow, and Caffe; deserialize the deep learning models of each training platform according to the model format of different platforms to obtain the operational characteristics of each node, node connection relationships and weight parameters of the model; convert the models of different formats into static graph models; and reserialize and save the model weight parameters.
[0063] The model analysis submodule is used to parse the static graph model; develop a static graph traversal method to obtain the connection relationships of each node in the static graph model, the correspondence between operator types and parameterized network nodes and weight parameters, based on the structure and node names of the static graph model; develop an operator size calculation library to calculate the size of each node's feature map based on the computational characteristics of different operators and the size of the input feature map; and develop an operator parsing library to obtain a complete set of parameters controlling the characteristics of each operation node from the static graph format data based on the characteristics of different operator operations, and reserve a parameter saving interface to facilitate the saving of characteristic parameter sets corresponding to different operator operations.
[0064] In this embodiment, the model intermediate representation module includes a model middleware storage submodule and a model middleware loading and access submodule;
[0065] The model middleware storage submodule is an embeddable module. First, the operation-level parameter descriptions in the model middle representation module are aligned with the characteristic parameter sets corresponding to each operator operation in the model parsing module. Then, the model middleware storage submodule is connected to the operator parsing library of the model parsing module. Finally, the static graph nodes are traversed using the static graph traversal method of the model parsing module, and the corresponding operator parsing library is called according to the operator type mapping. The connected model middleware storage submodule is used to automatically generate a general model middleware.
[0066] The model middleware loading and access submodule is used to complete the full loading of the model middleware, and to reassign parameters, replace operations, and merge operations in the model.
[0067] The model middleware loading and access submodule automatically generates middleware model loading code based on the rules of the multi-level description of the model, by maintaining a table corresponding to the loading methods of operator parameters and a table corresponding to the data types of operator parameters, and designing code generation rules. When adding a new operator, only the operation-level parameter description of the model's middleware description needs to be updated, and the loading method of the corresponding operator can be automatically generated. This greatly improves the development efficiency of the deployment system and ensures the robustness of model loading.
[0068] In the model access section, the model middleware loading and access submodule locates the operator nodes that need to be updated or deleted based on the network-level description of the middleware model network node number, node type, and node input-output connection relationship. It can also freely access and manipulate the characteristic parameters of the corresponding operator, and can reconstruct the intermediate representation model as needed, and save the reconstructed intermediate format model.
[0069] In this embodiment, the model deployment module includes a deployment platform interface library, a deployment platform operator conversion library, and a deployment platform hardware language supplementary library. The deployment platform operator conversion library and the deployment platform hardware language supplementary library enable top-down, multi-level access and development to various independently controllable chip deployment platforms, and complete the automatic deployment of models on different chip platforms.
[0070] The deployment platform interface library is used for accessing and developing development tools at different chip levels, including a deployment engine initialization module, a middleware model loading module, a model traversal and conversion module, and a deployment engine generation module. The deployment engine initialization module and deployment engine generation module related to the deployment engine are encapsulated and developed according to the development interfaces of different chip platforms.
[0071] The deployment engine initialization module is coupled with the method for generating deployment engine files for different chip platforms, and is used for predefining data structures and initializing special operations required for generating deployment engine files for different chip platforms.
[0072] The middleware model loading module automatically generates the model based on the intermediate representation module, and completes the loading and free access to the parameters of the intermediate format model.
[0073] The model traversal and conversion module, based on the characteristics of different chips, calls the deployment platform operator conversion library to convert the operators of the intermediate format model into high-speed computing operators supported by the corresponding deployment platform. During the traversal process, it maintains the connection relationship of the operations of each node of the deployment engine, and completes the conversion of the entire model to the target deployment platform format. If there are operators in the model that are not supported by the target deployment platform, they are marked and serialized for storage, which facilitates the subsequent development of custom operators.
[0074] The deployment engine generation module generates a deployment engine for the corresponding chip platform based on the successfully traversed deployment engine stream. The deployment engine generation module can modify the input and output nodes of the deployment engine stream to generate deployment engines corresponding to different continuous parts of the model, thereby completing model splicing and model segmentation, and improving the freedom of model deployment.
[0075] In this embodiment, the deployment platform operator conversion library is developed at the network model operator level, according to the different support of different chip development tools, and is corresponding to the chip platform. The deployment platform operator conversion library converts the general operators in the network-level parameter description, operation-level parameter description and operator parameter characteristic-level description into high-speed computing operators supported by the deployment platform, which is achieved through operator type mapping and operator parameter conversion.
[0076] The operator type mapping is based on the operator support of the deployment platform, maintaining a one-to-one correspondence between general operators and high-speed computing operators of the deployment platform; through the mapping table, the operator operations described by the intermediate format model during model traversal can automatically correspond to the high-speed computing operator interfaces of the same operations supported by the chip platform.
[0077] The operator parameter conversion involves converting the operator parameters of the intermediate format model according to the parameter interface characteristics of the high-speed computing operator on the chip platform, so that the parameters conform to the calling logic of the high-speed computing operator, and assigning values in the high-speed computing operator calling interface to complete the operator conversion.
[0078] In this embodiment, the deployment platform hardware language supplementary library is a supplementary library developed using the corresponding deployment platform hardware language at the deployment platform hardware language level, based on the different levels of chip openness. It is used to implement functions not officially supported by the chip platform, including custom operator development, model optimization, and efficient model pre- and post-processing development.
[0079] The development of custom operators involves analyzing the computational principles of unsupported operators based on the list of unsupported operators obtained from the model traversal module, writing corresponding computational operations using the hardware language of the chip platform, and then encapsulating the written operators into custom operators using chip development tools, adding them to the operator conversion library, and updating the operator mapping table.
[0080] Model optimization involves performing model quantization, operation merging, and operator optimization based on the hardware characteristics of different chips and their corresponding hardware languages, so that the deployed model can achieve better running performance on the corresponding domestically controlled chip.
[0081] High-efficiency model pre- and post-processing development involves developing efficient model pre- and post-processing operations using chip hardware languages based on the pre- and post-processing operation principles corresponding to different models. This enhances the model deployment capabilities of the corresponding chip deployment platform, reduces computational operations on non-chip devices, and improves deployment efficiency and performance.
[0082] This invention also provides an automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representations, employing the aforementioned automatic deployment system. The automatic deployment method includes:
[0083] (1) Load the deep learning model trained from the model training platform through the model parsing module, and parse the model network structure, operation type and operation parameters of each layer of the deep learning model;
[0084] (2) A general model intermediate representation method is defined by using the model intermediate representation module. The deep learning model parsed in step (1) is formatted and stored as a general intermediate format model through the model intermediate representation method.
[0085] (3) Using the model deployment module, traverse the intermediate format model saved in the intermediate format in step (2), and perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms to complete the automatic deployment of deep learning models on different independent and controllable chips.
[0086] In step (2) of this embodiment, the deep learning model parsed in step (1) is first described in a multi-level complete manner using the model intermediate representation method. Then, the model middleware storage submodule and the model middleware loading and access submodule are used to perform accurate model conversion, model quantization, model pruning, and model operation merging.
[0087] In this embodiment, the method of using the intermediate model representation to perform a multi-level complete description of the deep learning model parsed in step (1) includes:
[0088] Multi-level descriptions include network-level parameter descriptions, operation-level parameter descriptions, and operator parameter characteristic-level descriptions;
[0089] The network-level parameter description is the highest level of model description, including the size of the input feature map of the deep learning network, the connection relationship of each network node, the operator type of each network node, the storage sequence number of the network node, and the correspondence between weighted nodes and weight parameter files.
[0090] The operation-level parameter descriptions are intermediate layers in the model description, and are described separately for different operator operations; specifically, for example, the convolutional layer includes the convolution kernel size, convolution stride, and convolution dilation coefficient, and the activation function layer includes the activation function type, the corresponding Alpha and Beta parameters, etc.
[0091] The operator parameter characteristic level description is the lowest level of the model description, including the data type, precision, and whether the parameter is a sequence.
[0092] In this embodiment, in step (3), the deployment platform interface library, the deployment platform operator conversion library and the deployment platform hardware language supplementary library are used to realize the access and development of multiple independent and controllable chip deployment platforms from top to bottom and multi-level, and to complete the automatic deployment of the model on different chip platforms.
[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. An automated deployment system for an autonomous and controllable chip model based on multi-level intermediate representation, characterized in that, The automated deployment system includes: Model parsing module: used to load the deep learning model trained from the model training platform, and parse the model network structure, operation types and operation parameters of each layer of the deep learning model; Model intermediate representation module: This module defines a general model intermediate representation method, which is used to format and store the deep learning model parsed by the model parsing module into a general intermediate format model. Model deployment module: used to traverse the intermediate format model saved in intermediate format, perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms, and complete the automatic deployment of deep learning models on different self-controllable chips; The model parsing module includes a model loading submodule and a model analysis submodule; The model loading submodule is used to load model formats from different training platforms such as PyTorch, Tensorflow, and Caffe; deserialize the deep learning models of each training platform according to the model format of different platforms to obtain the operational characteristics of each node, node connection relationships and weight parameters of the model; convert the models of different formats into static graph models; and reserialize and save the model weight parameters. The model analysis submodule is used to parse the static graph model; develop a static graph traversal method to obtain the connection relationships of each node in the static graph model, the correspondence between operator types and parameterized network nodes and weight parameters, based on the structure and node names of the static graph model; develop an operator size calculation library to calculate the size of each node's feature map based on the computational characteristics of different operators and the size of the input feature map; and develop an operator parsing library to obtain a complete set of parameters controlling the characteristics of each operation node from the static graph format data based on the characteristics of different operator operations, and reserve a parameter saving interface to facilitate the saving of characteristic parameter sets corresponding to different operator operations.
2. The automatic deployment system for an autonomous and controllable chip model based on multi-level intermediate representation as described in claim 1, characterized in that, The model intermediate representation module includes a model middleware storage submodule and a model middleware loading and access submodule; The model middleware storage submodule is an embeddable module; firstly, the operation-level parameter descriptions in the model middle representation module are aligned with the characteristic parameter sets corresponding to each operator operation in the model parsing module; Then, the model middleware storage submodule is connected to the operator parsing library of the model parsing module; finally, the static graph nodes are traversed through the static graph traversal method of the model parsing module, and the corresponding operator parsing library is called according to the operator type mapping, and the general model middleware is automatically generated using the connected model middleware storage submodule. The model middleware loading and access submodule is used to complete the full loading of the model middleware, and to reassign parameters, replace operations, and merge operations in the model. The model middleware loading and access submodule automatically generates middleware model loading code based on the rules of the multi-level description of the model, by maintaining the correspondence table of operator parameter loading methods and the correspondence table of operator parameter data types, and designing code generation rules. When adding a new operator, it is only necessary to update the operation-level parameter description of the model middleware description, and the corresponding operator loading method can be automatically generated. The model middleware loading and access submodule locates the operator nodes that need to be updated or deleted based on the network-level description of the network node number, node type, and node input-output connection relationship of the middleware model to be accessed. It can also freely access and manipulate the characteristic parameters of the corresponding operators, reconstruct the intermediate representation model as needed, and save the reconstructed intermediate format model.
3. The automatic deployment system for an autonomous and controllable chip model based on multi-level intermediate representation as described in claim 2, characterized in that, The model deployment module includes a deployment platform interface library, a deployment platform operator conversion library, and a deployment platform hardware language supplementary library; The deployment platform interface library is used for accessing and developing development tools at different chip levels, including a deployment engine initialization module, a middleware model loading module, a model traversal and conversion module, and a deployment engine generation module. The deployment engine initialization module and deployment engine generation module related to the deployment engine are encapsulated and developed according to the development interfaces of different chip platforms. The deployment engine initialization module is coupled with the method for generating deployment engine files for different chip platforms, and is used for predefining data structures and initializing special operations required for generating deployment engine files for different chip platforms. The middleware model loading module automatically generates the model based on the intermediate representation module, and completes the loading and free access to the parameters of the intermediate format model. The model traversal and conversion module, based on the characteristics of different chips, calls the deployment platform operator conversion library to convert the operators of the intermediate format model into high-speed computing operators supported by the corresponding deployment platform. During the traversal process, it maintains the connection relationship of the operations of each node of the deployment engine, and completes the conversion of the entire model to the target deployment platform format. If there are operators in the model that are not supported by the target deployment platform, they are marked and serialized for storage, which facilitates the subsequent development of custom operators. The deployment engine generation module generates a deployment engine for the corresponding chip platform based on the successfully traversed deployment engine stream; The deployment engine generation module can modify the input and output nodes of the deployment engine flow to generate deployment engines corresponding to different continuous parts of the model, thereby completing model splicing and model segmentation, and improving the freedom of model deployment.
4. The automatic deployment system for an autonomous and controllable chip model based on multi-level intermediate representation according to claim 3, characterized in that, The deployment platform operator conversion library is developed at the network model operator level, according to the different support of different chip development tools, and is corresponding to the chip platform. The deployment platform operator conversion library converts the general operators in the network-level parameter description, operation-level parameter description and operator parameter characteristic-level description into high-speed computing operators supported by the deployment platform, which is achieved through operator type mapping and operator parameter conversion. The operator type mapping is based on the operator support of the deployment platform, maintaining a one-to-one correspondence between general operators and high-speed computing operators of the deployment platform; through the mapping table, the operator operations described by the intermediate format model during model traversal can automatically correspond to the high-speed computing operator interfaces of the same operations supported by the chip platform. The operator parameter conversion involves converting the operator parameters of the intermediate format model according to the parameter interface characteristics of the high-speed computing operator on the chip platform, so that the parameters conform to the calling logic of the high-speed computing operator, and assigning values in the high-speed computing operator calling interface to complete the operator conversion.
5. The automatic deployment system for an autonomous and controllable chip model based on multi-level intermediate representation according to claim 3, characterized in that, The deployment platform hardware language supplementary library is developed at the deployment platform hardware language level, based on the different levels of chip openness, using the corresponding deployment platform hardware language to develop supplementary libraries for the chip platform; it is used to implement functions not officially supported by the chip platform, including custom operator development, model optimization, and efficient model pre- and post-processing development; The development of custom operators involves analyzing the computational principles of unsupported operators based on the list of unsupported operators obtained from the model traversal module, writing corresponding computational operations using the hardware language of the chip platform, and then encapsulating the written operators into custom operators using chip development tools, adding them to the operator conversion library, and updating the operator mapping table. Model optimization involves performing model quantization, operation merging, and operator optimization based on the hardware characteristics of different chips and their corresponding hardware languages. High-efficiency model pre- and post-processing development involves developing efficient model pre- and post-processing operations using chip hardware languages, based on the pre- and post-processing operation principles corresponding to different models.
6. An automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation, wherein the automatic deployment method employs the automatic deployment system described in any one of claims 1-5, characterized in that, The automatic deployment method includes: (1) Load the deep learning model trained from the model training platform through the model parsing module, and parse the model network structure, operation type and operation parameters of each layer of the deep learning model; (2) A general model intermediate representation method is defined by using the model intermediate representation module. The deep learning model parsed in step (1) is formatted and stored as a general intermediate format model through the model intermediate representation method. (3) Using the model deployment module, traverse the intermediate format model saved in the intermediate format in step (2), and perform corresponding operator conversion, model optimization and deployment model file generation operations for different chip platforms to complete the automatic deployment of deep learning models on different independent and controllable chips.
7. The automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation according to claim 6, characterized in that, In step (2), the deep learning model parsed in step (1) is first described in a multi-level complete manner using the model intermediate representation method. Then, the model middleware storage submodule and the model middleware loading and access submodule are used to perform precise model conversion, model quantization, model pruning, and model operation merging.
8. The automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation according to claim 7, characterized in that, The method of using the intermediate representation of the model to perform a multi-level complete description of the deep learning model parsed in step (1) includes the following specific methods: Multi-level descriptions include network-level parameter descriptions, operation-level parameter descriptions, and operator parameter characteristic-level descriptions; The network-level parameter description is the highest level of model description, including the size of the input feature map of the deep learning network, the connection relationship of each network node, the operator type of each network node, the storage sequence number of the network node, and the correspondence between weighted nodes and weight parameter files. The operation-level parameter description is an intermediate layer in the model description, and is described separately for different operator operations; The operator parameter characteristic level description is the lowest level of the model description, including the data type, precision, and whether the parameter is a sequence.
9. The automatic deployment method for an autonomous and controllable chip model based on multi-level intermediate representation according to claim 6, characterized in that, In step (3), the deployment platform interface library, the deployment platform operator conversion library and the deployment platform hardware language supplementary library are used to realize the access and development of multiple independent and controllable chip deployment platforms from top to bottom and multi-level, and to complete the automatic deployment of the model on different chip platforms.
Citation Information
Patent Citations
Model deployment method based on artificial intelligence platform and related equipment
CN114861836A
