Image recognition model deployment method and system

Abstract

The invention belongs to the technical field of algorithm models, and particularly relates to an image recognition model deployment method and system. In order to overcome the defect that an existing model deployment method cannot or cannot well deploy a model to different chips, the invention adopts the following technical scheme: the image recognition model deployment method comprises the following steps: converting a model to be deployed into a standard model in a uniform format; loading the standard model in a to-be-deployed chip environment, analyzing and restoring to obtain all information of the to-be-deployed model; judging whether the to-be-deployed chip supports network segmentation or not; after converting operators supported by the to-be-deployed chip, compiling the operators by adopting a compiling tool of a to-be-deployed chip manufacturer to obtain a hard computing result, and computing the operators which are not supported by the to-be-deployed chip by adopting a processor; and inputting the hard computing result into a reasoning tool of a to-be-deployed chip manufacturer, and inputting to-be-reasoned data into the reasoning tool or a processor for reasoning to obtain a reasoning result. The method has the beneficial effect that the deployment of the same algorithm model on different chips is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of algorithm models, and specifically relates to an image recognition model deployment method and system. Background Art

[0002] In the current artificial intelligence industry, there are various types of chips used to carry image recognition models, including NVIDIA, Huawei, Cambrian, Rockchip, Horizon, etc. Each type of chip has different requirements for algorithm interfaces and is incompatible with each other, resulting in the binding of algorithms and chips. When replacing chips, algorithms need to be redeveloped, and when updating algorithms, only existing chips can be selected. This fragmented phenomenon of mutual incompatibility directly results in high application development costs for algorithms, indirectly leads to the non-openness of the artificial intelligence field, reduces the innovation motivation of algorithm developers, and ultimately hinders the practical application of artificial intelligence algorithms.

[0003] CN114880995A discloses an algorithm solution deployment method and related devices, equipment and storage media, wherein the algorithm solution deployment method includes: in response to a newly created solution project on a solution platform, based on the algorithm model library built into the solution platform, obtaining a number of first algorithm models applicable to the solution project, and based on the conversion tool library built into the solution platform, obtaining a format conversion tool applicable to a target platform; wherein the target platform represents a chip platform deployed after the solution project is constructed; based on the format conversion tool, format conversion is performed on a number of first algorithm models respectively to obtain a second algorithm model matching the target platform; and the solution is deployed based on the second algorithm model to obtain an algorithm solution.

[0004] The above-mentioned solution can realize the deployment and application of algorithm solutions on various chip platforms. However, there are still some problems as follows: 1. It realizes the development of algorithms from scratch, and cannot deploy the developed algorithms on various chip platforms; 2. The built-in algorithm model library needs to be configured with models that adapt to various chip platforms, and the conversion tool library needs to be configured with format conversion tools suitable for various chip platforms; 3. Models can only be selected from the built-in algorithm model library; 4. For various chip platforms, the same complete process needs to be followed, that is, different chip platforms need to be redeveloped. Summary of the invention

[0005] The present invention aims at the insufficiency that the existing model deployment method cannot or cannot well implement the model deployment to different chips, and provides an image recognition model deployment method to implement the deployment of the developed algorithm on various chip platforms. The present invention also provides an image recognition model deployment system.

[0006] To achieve the above object, the present invention adopts the following technical solution: an image recognition model deployment method, the image recognition model deployment method comprising:

[0007] Step S1: convert the model to be deployed into a standard model in a unified format, where the standard model retains all information of the model to be deployed;

[0008] Step S2: Load the standard model in the chip environment to be deployed, and restore all the information of the model to be deployed after parsing;

[0009] Step S3: query the operator information database of the chip manufacturer to be deployed, and divide the network according to whether the chip to be deployed supports it;

[0010] Step S4: compile the operators supported by the chip to be deployed using the compilation tool of the chip manufacturer to obtain executable content;

[0011] Step S5: input the executable content into the reasoning tool of the chip manufacturer to be deployed, input the data to be inferred into the reasoning tool or processor for reasoning, and obtain the reasoning result.

[0012] As an improvement, in step S1, nodes are used to represent operators of the model to be deployed, and all information of the model to be deployed includes node names, connection relationships between nodes, and input and output of nodes.

[0013] As an improvement, in step S1, the development framework of the model to be deployed is identified through the model conversion tool, and the tensors and node names of each layer of the model to be deployed are described using the same rules.

[0014] As an improvement, in step S1, the model structure is optimized by a model conversion tool, and the optimization method includes:

[0015] Operator fusion is to fuse two or more operators that can be combined into one operator.

[0016] As an improvement, in step S1, a model conversion tool is used to serialize the standard model to output the node names, the connection relationships between nodes, and the input and output of the nodes.

[0017] As an improvement, in step S2, a deserialization module is used to parse the sequence of the standard model and extract the parameters required for the calculation of each layer, including tensors, data type arrangements, and operators.

[0018] As an improvement, in step S3, the deserialized standard model is layered and labeled, and a compilation tool or processor is selected for different layers according to the labels;

[0019] In step S4, according to the contents of the operator information library, the correct operator in the operator pool is called to perform layer-by-layer calculations, and after compilation, a handle representing the network structure is obtained;

[0020] In step S4, according to the contents of the operator information library, multiple operators in the operator pool are called at one time and fusion compilation is completed in advance, and the fused operators are used for calculation;

[0021] In step S5, the inference result is represented by a matrix, and the matrix contains complete calculation result information.

[0022] As an improvement, the image recognition model deployment method further includes:

[0023] Step S6: Release the reasoning resources.

[0024] An image recognition model deployment system adopts the aforementioned image recognition model deployment method, and the image recognition model deployment system includes:

[0025] Model conversion tool, used to convert the model to be deployed into a standard model in a unified format. The standard model contains all the information of the algorithm to be deployed;

[0026] The model parsing module is used to load the standard model in the chip environment to be deployed, and restore all the information of the standard model after parsing;

[0027] Operator information library module, used to store operators of various chip manufacturers;

[0028] The query and segmentation module is used to query the operator information database of the chip manufacturer to be deployed, and segment the network according to whether the chip to be deployed supports it;

[0029] The network compilation module is used to compile the operators supported by the chips to be deployed using the compilation tools of the chip manufacturer to obtain executable content;

[0030] The network computing module is used to input the executable content into the inference tool of the chip manufacturer to be deployed, input the data to be inferred into the inference tool or processor for inference, and obtain the inference result;

[0031] The resource release module is used to release the inference resources after the inference is completed.

[0032] As an improvement of the image recognition model deployment system, the model conversion tool includes a parsing module, an operator fusion module and a serialization module;

[0033] The model parsing module, operator information library module, query segmentation module, network compilation module and network computing module are integrated in the SDK.

[0034] The image recognition model deployment method of the present invention converts the model to be deployed into a standard model in a unified format, the standard model retains all information of the model to be deployed, loads the standard model in the chip environment to be deployed, and restores all information of the model to be deployed after parsing, queries the operator information library of the chip manufacturer to be deployed, and compiles the operators supported by the chip to be deployed using the compilation tool of the chip manufacturer to be deployed according to whether the chip to be deployed supports network segmentation to obtain executable content, inputs the executable content into the reasoning tool of the chip manufacturer to be deployed, inputs the data to be inferred into the reasoning tool or processor for reasoning to obtain the reasoning result, and through the above steps, the developed algorithm model can be deployed on various chip platforms; the algorithm model can be developed under various frameworks with a large degree of freedom; and some steps of deploying the same algorithm model to different chip platforms only need to be done once. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural block diagram of a model conversion tool of an image recognition model deployment system according to an embodiment of the present invention.

[0036] Figure 2 It is a structural block diagram of the SDK (Software Development Kit) of the image recognition model deployment system of an embodiment of the present invention.

[0037] Figure 3 It is a flowchart of the image recognition model deployment method according to an embodiment of the present invention.

[0038] Figure 4 It is a model structure diagram obtained after a model conversion tool of the image recognition model deployment system of an embodiment of the present invention reads a model.

[0039] Figure 5 It is a model structure diagram obtained after the model conversion tool of the image recognition model deployment system of an embodiment of the present invention fuses the operators.

[0040] Figure 6 It is a model serialization representation diagram output by the model conversion tool of the image recognition model deployment system in an embodiment of the present invention after the conversion is completed.

[0041] Figure 7 It is a diagram of the segmented model structure of the image recognition model deployment system according to an embodiment of the present invention.

[0042] Figure 8 It is a compiled model structure diagram of the image recognition model deployment system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions of the embodiments of the present invention are explained and described below, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0044] See also Figures 1 to 8 , the image recognition model deployment system of the embodiment of the present invention, the image recognition model deployment system comprises:

[0045] Model conversion tool, used to convert the model to be deployed into a standard model in a unified format. The standard model contains all the information of the algorithm to be deployed;

[0046] The model parsing module is used to load the standard model in the chip environment to be deployed, and restore all the information of the standard model after parsing;

[0047] Operator information library module, used to store operators of various chip manufacturers;

[0048] The query and segmentation module is used to query the operator information database of the chip manufacturer to be deployed, and segment the network according to whether the chip to be deployed supports it;

[0049] The network compilation module is used to compile the operators supported by the chips to be deployed using the compilation tools of the chip manufacturer to obtain executable content;

[0050] The network computing module is used to input the executable content into the inference tool of the chip manufacturer to be deployed, input the data to be inferred into the inference tool or processor for inference, and obtain the inference result;

[0051] The resource release module is used to release the inference resources after the inference is completed.

[0052] See also Figure 1 and Figure 2 In this embodiment, the image recognition model deployment system includes two main parts: a model conversion tool and an SDK. The model conversion tool includes a parsing module, an operator fusion module, and a serialization module. The model parsing module, the operator information library module, the query segmentation module, the network compilation module, and the network calculation module are integrated in the SDK.

[0053] In this embodiment, the entire image recognition model deployment system is provided to users (such as algorithm developers). Users deploy the SDK on hardware devices in the form of a library to provide an environment for the operation of the converted model.

[0054] In this embodiment, the parsing module of the model conversion tool has parsing submodules for various development frameworks such as caffe, tensorflow, darkent, etc. Through the parsing module, it can be known based on which development framework the received algorithm model is developed.

[0055] In this embodiment, the operator fusion module of the model conversion tool can fuse nodes with consistent calculation forms, thereby greatly accelerating the reasoning speed.

[0056] In this embodiment, the model conversion tool uses a serialized approach to express the model.

[0057] In this embodiment, the deserialization module of the SDK performs parsing to obtain all standardized information. Standardization means that the algorithm models developed by each development framework use the same expression method.

[0058] In this embodiment, the network segmentation module of the SDK queries the operator information library and segments the network according to whether the chip to be deployed supports it.

[0059] In this embodiment, after the network is segmented, the network compilation module calls the compilation tool of the chip manufacturer to separately compile the supported operators. The compilation result is binary machine code content, which is stored in the memory to form a processor (such as CPU, GPU) computing network for unsupported operators.

[0060] In this embodiment, after the compilation is completed, the network calculation module performs calculations. Specifically, the inference tool of the chip manufacturer is called to calculate supported operators. The processor is used to calculate unsupported operators.

[0061] In this embodiment, after the reasoning is completed, the resource release module releases the reasoning resources. Users (such as algorithm developers) can extract information from the calculation result matrix according to their needs and perform subsequent operations.

[0062] In this embodiment, the operator fusion module of the model conversion tool is used to fuse some operators in the model to obtain a standard model. The model conversion tool has an operator fusion function (these operators are uniformly represented by nodes). When the calculation forms of multiple operators are consistent, they can be fused. The optimized model structure can greatly speed up the reasoning speed. After conversion, the second algorithm model is saved as a file.

[0063] See also Figures 3 to 8 , an image recognition model deployment method, the image recognition model deployment method comprising:

[0064] Step S1: convert the model to be deployed into a standard model in a unified format, where the standard model retains all information of the model to be deployed;

[0065] Step S2: Load the standard model in the chip environment to be deployed, and restore all the information of the model to be deployed after parsing;

[0066] Step S3: query the operator information database of the chip manufacturer to be deployed, and divide the network according to whether the chip to be deployed supports it;

[0067] Step S4: Compile the operators supported by the chip to be deployed using the compilation tool of the chip manufacturer to obtain executable content; for operators not supported, there is no compilation step, and the processor is directly used for calculation processing in subsequent steps;

[0068] Step S5: input the executable content into the reasoning tool of the chip manufacturer to be deployed, input the data to be inferred into the reasoning tool or processor for reasoning, and obtain the reasoning result.

[0069] In this embodiment, in step S1, nodes are used to represent operators of the model to be deployed, and all information of the model to be deployed includes node names, connection relationships between nodes, and input and output of nodes.

[0070] In this embodiment, in step S1, the development framework of the model to be deployed is identified by a model conversion tool, and the tensors and node names of each layer of the model to be deployed are described using the same rules.

[0071] In this embodiment, in step S1, the model structure is optimized by a model conversion tool, and the optimization method includes:

[0072] Operator fusion is to fuse two or more operators that can be combined into one operator.

[0073] In this embodiment, in step S1, a model conversion tool is used to serialize the standard model to output the node names, the connection relationships between nodes, and the input and output of the nodes.

[0074] In this embodiment, in step S2, a deserialization module is used to parse the sequence of the standard model to extract various parameters required for calculation of each layer, and the parameters include tensors, data type arrangements, and operators.

[0075] In this embodiment, in step S3, the deserialized standard model is layered and marked, and a compilation tool or processor is selected for different layers according to the mark;

[0076] In step S4, according to the contents of the operator information library, the correct operator in the operator pool is called to perform layer-by-layer calculations, and after compilation, a handle representing the network structure is obtained;

[0077] In step S4, according to the content of the operator information library, multiple operators in the operator pool are called at one time and fusion compilation is completed in advance, and the fused operators are used for calculation.

[0078] In step S4, the supported operators are converted before compilation. The conversion is to represent all the information of the operator (including input information, output information, arrangement information, operator name, etc.) in a data structure according to the requirements of the chip manufacturer, and then start compilation.

[0079] In step S5, the inference result is represented by a matrix, and the matrix contains complete calculation result information.

[0080] The following combination Figures 1 to 8 The process of the deployment method of the embodiment of the present invention is described in more detail.

[0081] Assume that the structure of an image recognition model to be deployed is as follows Figure 4 As shown in the figure, the original model is trained by the caffe framework. The image recognition model has 6 nodes (nodes, used to represent operators), and Tensor0-7 represent different tensors. After Tensor0 is input into Node0, it outputs Tensor1 to Node1 and outputs Tensor to Node2. Node1 outputs Tensor3 to Node3, Node2 outputs Tensor4 to Node3, Node3 outputs Tensor5 to Node4, Node4 outputs Tensor6 to Node5, and finally Node5 outputs Tensor7.

[0082] First, use the model conversion tool to Figure 4 The original model is converted to the standard model. The main structure of the model conversion tool can be found in Figure 1 The parsing module of the model conversion tool has a caffe submodule corresponding to the caffe framework, and the model network information of the original model (set as the first model file) is read through the caffe submodule to obtain Figure 4 information.

[0083] The model conversion tool has the function of operator fusion (these operators are uniformly represented by nodes). Since the calculation forms of node4 and node5 are the same, they can be fused to obtain Figure 5 The model structure shown in Figure 1. The optimized model structure can significantly speed up the inference speed.

[0084] The model conversion tool has a serialization module, and the converted model is presented in a serialized form (see Figure 6 ), and save it as a file, and set it as the second model file. The second model file is also the file input to the SDK.

[0085] Then, the SDK parses the content of the second model file through the internal deserialization module, and obtains all the information of the standard model after parsing.

[0086] Secondly, the SDK queries the operator information library of the chip manufacturer to be deployed through the internal network segmentation module, and segments the network according to whether the chip to be deployed supports it. In this example, Node0, Node1, Node2, and Node3 are supported by the AI ​​chip, but Node4 is not supported. The sub-network will be segmented into Figure 7 Condition.

[0087] Once the network is segmented, the SDK calls the chip manufacturer's compilation tool through the internal network compilation module to compile the supported operators separately. The compilation result is binary machine code content, which is stored in the memory. Figure 8 As shown, the chip computing network composed of node0, node1, node2, and node3 has been compiled into a machine code content that can be directly run. For the unsupported Node4, a CPU computing network is formed, and the CPU is used for computing.

[0088] Finally, each sub-network is calculated. The chip computing network is calculated first. The calculation method for this network is to call the inference tool of the chip manufacturer to perform calculations. The calculation results are given to the CPU computing network, that is, node4. The calculation process of node4 is implemented on the CPU. Finally, the matrix result stored in the form of tensor6 is obtained.

[0089] After the inference is completed, the inference resources are released. Users (such as algorithm developers) can extract information from the calculation result matrix according to their needs and perform subsequent operations.

[0090] When the same algorithm model needs to be deployed to different chip platforms, just repeat steps S2 to S6 according to the corresponding chip.

[0091] The above is only a specific implementation of the invention, but the protection scope of the invention is not limited thereto. Those skilled in the art should understand that the invention includes but is not limited to the contents described in the above specific implementation. Any modification that does not deviate from the functional and structural principles of the invention will be included in the scope of the claims.

Claims

1. An image recognition model deployment method, characterized in that: The image recognition model deployment method comprises: Step S1: convert the model to be deployed into a standard model in a unified format, where the standard model retains all information of the model to be deployed; Step S2: Load the standard model in the chip environment to be deployed, and restore all the information of the model to be deployed after parsing; Step S3: query the operator information database of the chip manufacturer to be deployed, and divide the operator according to whether the chip to be deployed supports it; Step S4: compile the operators supported by the chip to be deployed using the compilation tool of the chip manufacturer to obtain executable content; Step S5: input the executable content into the reasoning tool of the chip manufacturer to be deployed, input the data to be inferred into the reasoning tool or processor for reasoning, and obtain the reasoning result.

2. The image recognition model deployment method according to claim 1, characterized in that: In step S1, nodes are used to represent operators of the model to be deployed. All information of the model to be deployed includes node names, connection relationships between nodes, and input and output of nodes.

3. The image recognition model deployment method according to claim 2, characterized in that: In step S1, the development framework of the model to be deployed is identified through the model conversion tool, and the tensors and node names of each layer of the model to be deployed are described using the same rules.

4. The image recognition model deployment method according to claim 3, characterized in that: In step S1, the model structure is optimized by a model conversion tool, and the optimization method includes: Operator fusion is to fuse two or more operators that can be combined into one operator.

5. The image recognition model deployment method according to claim 4, characterized in that: In step S1, a model conversion tool is used to serialize the standard model to output the node names, the connection relationships between nodes, and the input and output of the nodes.

6. The image recognition model deployment method according to claim 5, characterized in that: In step S2, the sequence of the standard model is parsed using a deserialization module to extract the parameters required for calculation of each layer, including tensors, data type arrangements, and operators.

7. The image recognition model deployment method according to claim 6, characterized in that: In step S3, the deserialized standard model is layered and labeled, and a compilation tool or processor is selected for different layers according to the labels; In step S4, before compiling, the supported operators are represented in data structure according to the requirements of the chip manufacturer; In step S4, according to the contents of the operator information library, the correct operator in the operator pool is called to perform layer-by-layer calculations, and after compilation, a handle representing the network structure is obtained; In step S4, according to the contents of the operator information library, multiple operators in the operator pool are called at one time and fusion compilation is completed in advance, and the fused operators are used for calculation; In step S5, the inference result is represented by a matrix, and the matrix contains complete calculation result information.

8. The image recognition model deployment method according to claim 1, characterized in that: The image recognition model deployment method also includes: Step S6: Release the reasoning resources.

9. An image recognition model deployment system, using the image recognition model deployment method according to any one of claims 1 to 8, characterized in that: The image recognition model deployment system comprises: Model conversion tool, used to convert the model to be deployed into a standard model in a unified format. The standard model contains all the information of the algorithm to be deployed; The model parsing module is used to load the standard model in the chip environment to be deployed, and restore all the information of the standard model after parsing; Operator information library module, used to store operators of various chip manufacturers; The query and segmentation module is used to query the operator information database of the chip manufacturer to be deployed, and segment the operators according to whether the chip to be deployed supports it; The network compilation module is used to compile the operators supported by the chips to be deployed using the compilation tools of the chip manufacturer to obtain executable content; The network computing module is used to input the executable content into the inference tool of the chip manufacturer to be deployed, input the data to be inferred into the inference tool or processor for inference, and obtain the inference result; The resource release module is used to release the inference resources after the inference is completed.

10. The image recognition model deployment system according to claim 9, characterized in that: The model conversion tool includes a parsing module, an operator fusion module and a serialization module; The model parsing module, operator information library module, query segmentation module, network compilation module and network computing module are integrated in the SDK.

Images