A network model deployment method, device, storage medium and edge device
By filtering operators not supported by NPUs in edge devices and using them as a breaking point for model splitting, the incomplete model deployment problem caused by limited computing power of NPUs is solved, and the high-accuracy deployment of network models is achieved.
Patent Information
- Application Number
- CN202111116953.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-09-23
AI Technical Summary
In edge computing devices, the computing power of NPUs is limited, resulting in the inability to effectively deploy complex network models containing NPUs that do not support operators, affecting the accuracy of the model.
By filtering out operators not supported by NPU as the boundary point, splitting the network model, deploying the sub-model on the NPU, and deploying the unsupported operators on the CPU, thereby achieving the complete deployment of the network model.
Ensure the complete deployment and high accuracy of the network model, and avoid the incomplete deployment of the model due to operator limitations.
Smart Images

Figure CN113806095B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network models, and specifically, to a network model deployment method, apparatus, storage medium and edge device. Background Art
[0002] With the development of edge computing, more and more edge computing devices can directly execute complex model reasoning processes, thereby saving the time of uploading data to cloud devices and downloading data to user devices. Compared with cloud devices, the computing power of the NPU of edge devices is limited, and the operators they support are limited.
[0003] Therefore, how to complete the deployment of the model to the edge device while ensuring high accuracy has become a difficult problem that plagues technicians in this field. Summary of the invention
[0004] The purpose of this application is to provide a network model deployment method, apparatus, storage medium and edge device to at least partially improve the above-mentioned problems.
[0005] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, an embodiment of the present application provides a network model deployment method, the method comprising:
[0007] According to the operators supported by the NPU in the edge device, filter out the operators not supported by the NPU in the network model;
[0008] Using the unsupported operator as a dividing point, splitting the network model to obtain at least two sub-models;
[0009] The sub-model is deployed on the NPU, and the unsupported operator is deployed on the CPU.
[0010] In a second aspect, an embodiment of the present application provides a network model deployment device, the device comprising:
[0011] A processing unit, configured to filter out operators not supported by the NPU in the network model according to operators supported by the NPU in the edge device;
[0012] The processing unit is further configured to split the network model using the unsupported operator as a dividing point to obtain at least two sub-models;
[0013] A deployment unit is used to deploy the sub-model on the NPU and deploy the unsupported operator on the CPU.
[0014] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor.
[0015] In a fourth aspect, an embodiment of the present application provides an edge device, comprising: a processor and a memory, wherein the memory is used to store one or more programs; when the one or more programs are executed by the processor, the above method is implemented.
[0016] Compared with the prior art, the network model deployment method, device, storage medium and edge device provided in the embodiments of the present application screen out the operators not supported by the NPU in the network model based on the operators supported by the NPU in the edge device; split the network model using the unsupported operators as the dividing point to obtain at least two sub-models; deploy the sub-models on the NPU, and deploy the unsupported operators on the CPU. The method provided in the embodiments of the present application can fully deploy the network model to the edge device even if the network model includes operators not supported by the NPU. Compared with limiting the use of specific operators to adapt to the NPU of the edge device, this method can ensure the completeness of the network model deployment, so that the deployed network model has a higher accuracy.
[0017] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 A schematic diagram of the structure of an edge device provided in an embodiment of the present application;
[0020] Figure 2 A flowchart of a network model deployment method provided in an embodiment of the present application;
[0021] Figure 3 A schematic diagram of network model splitting provided in an embodiment of the present application;
[0022] Figure 4 A schematic diagram of sub-steps of S103 provided in an embodiment of the present application;
[0023] Figure 5 One of the sub-step schematic diagrams of S103 provided in an embodiment of the present application;
[0024] Figure 6 One of the flowcharts of the network model deployment method provided in the embodiment of the present application;
[0025] Figure 7 A schematic diagram of the sub-steps of S101 provided in an embodiment of the present application;
[0026] Figure 8 A unit diagram of a network model deployment device provided in an embodiment of the present application.
[0027] In the figure: 10 - processor; 11 - memory; 12 - bus; 13 - communication interface; 201 - processing unit; 202 - deployment unit. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0029] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0031] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0032] In the description of the present application, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0033] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "disposed" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0034] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0035] The prior art completes the deployment of network models to edge devices by avoiding the use of operators that are not supported by the NPU and operators with high complexity in the design of the network model. However, due to the restrictions on the use of specific operators, it cannot be guaranteed that the network model can achieve a high degree of accuracy. For example, the deployment of a point cloud obstacle detection model to the NPU has many restrictions, and it cannot be guaranteed that the converted model can achieve a high degree of accuracy. In order to overcome the above problems, an embodiment of the present application provides a network model deployment method.
[0036] The embodiment of the present application provides an edge device, which may be a portable mobile terminal, a computer, a server, or other devices. Figure 1, a schematic diagram of the structure of an edge device. The edge device includes a processor 10, a memory 11, and a bus 12. The processor 10 and the memory 11 are connected via the bus 12, and the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.
[0037] The processor 10 may be an integrated circuit chip having the ability to process signals. In the implementation process, each step of the network model deployment method may be completed by an integrated logic circuit of hardware in the processor 10 or instructions in the form of software. The above-mentioned processor 10 may include a central processing unit (CPU); it may also include a network processor (NPU). Of course, there may be other types of processors, such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0038] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0039] The bus 12 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. Figure 1 Although only one bidirectional arrow is used in the figure, it does not mean that there is only one bus 12 or only one type of bus 12 .
[0040] The memory 11 is used to store programs, such as programs corresponding to the network model deployment device. The network model deployment device includes at least one software function module that can be stored in the memory 11 in the form of software or firmware or solidified in the operating system (OS) of the edge device. After receiving the execution instruction, the processor 10 executes the program to implement the network model deployment method.
[0041] Possibly, the edge device provided in the embodiment of the present application further includes a communication interface 13. The communication interface 13 is connected to the processor 10 via a bus. The edge device can interact with other terminals via the communication interface 13.
[0042] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of a portion of the edge device. The edge device may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0043] The network model deployment method provided in the embodiment of the present application can be applied to but not limited to Figure 1 For the edge devices shown, please refer to Figure 2 , the network model deployment method includes: S101, S102, S103 and S104.
[0044] S101, based on the operators supported by the NPU in the edge device, filter out the operators not supported by the NPU in the network model.
[0045] As mentioned above, the computing power of the NPU is limited, and some specific operators in complex network models may exceed the support range of the NPU. In order to complete this type of network model deployment, it is often necessary to simplify the network model and limit the use of specific operators, resulting in reduced accuracy in affecting the operation of the network model. In this application, there is no need to simplify the network model, and the network model can be fully deployed in the edge device.
[0046] It can be understood that the operator support description of the NPU includes the types, structures or names of all operators that the NPU can support. Through the operator support description, all operators that the NPU can support can be counted or filtered out. On the premise of knowing the operators supported by the NPU in the edge device, the operators that the NPU does not support in the network model can be filtered out by traversing each operator in the network model.
[0047] It should be noted that operators are specific model units, and algorithms are a way of thinking. The relationship between the two can be summarized as follows: operators are the medium for algorithm implementation; algorithms are the soul of operator application.
[0048] It can be understood that the network model can be a point cloud-based all-terrain obstacle detection model, an object recognition model, and other complex computing models.
[0049] S102, using the unsupported operator as a dividing point to split the network model to obtain at least two sub-models.
[0050] The sub-model includes at least one operator supported by the NPU.
[0051] Please refer to Figure 3 , Assume that the network model includes 14 operators, and the M operator and the N operator are not supported by the NPU. With the M operator and the N operator as the dividing point, the sub-model 1 and the sub-model 2 are obtained. It should be noted that the NPU can support all operators in the sub-model 1 and the sub-model 2.
[0052] Figure 3 The sub-model 1 and sub-model 2 shown in FIG. 1 both include multiple operators, but the number of operators in the sub-model is not limited here, and the sub-model may include only one operator. When the sub-model includes multiple operators, the multiple operators are continuous multiple operators. Figure 3 As shown, multiple operators in sub-model 1 and sub-model 2 are continuously connected.
[0053] It can be understood that the sub-model is split from the original network model, so the specific structure of the sub-model can be determined only through the input and output of the sub-model. The operators before and after the unsupported operator are the output of the previous sub-model and the input of the next sub-model. Therefore, as long as the position of the unsupported operator is located, the sub-model structure can be clarified and split. The unsupported operator operation is implemented through CPU programming to complete the migration, that is, the unsupported operator can be deployed on the CPU.
[0054] S103, deploy the sub-model on the NPU.
[0055] Please continue to refer to Figure 3 , NPU can support all operators in sub-model 1 and sub-model 2, so sub-model 1 and sub-model 2 can be deployed on NPU.
[0056] S104: The operator is not supported to be deployed on the CPU.
[0057] Optionally, the CPU may be a processor 10 in an edge device.
[0058] By deploying sub-models on the NPU and unsupported operators on the CPU, the limitations of the edge device's NPU on the network model can be ignored. Even if the network model includes operators that the NPU cannot support, it can be solved by splitting the model and deploying them separately, so that the complex network model can be fully deployed on the edge device, ensuring the completeness of the network model deployment, so that the deployed network model has a higher accuracy.
[0059] It can be understood that the CPU can call the NPU to execute the calculation process corresponding to the sub-model in a sequential relationship, thereby completing the network model calculation and completing the deployment of the network model to the edge device while ensuring a high degree of accuracy.
[0060] In summary, the embodiment of the present application provides a network model deployment method, which filters out operators that are not supported by the NPU in the network model based on the operators supported by the NPU in the edge device; splits the network model using the unsupported operators as the dividing point to obtain at least two sub-models; deploys the sub-models on the NPU, and deploys the unsupported operators on the CPU. The method provided in the embodiment of the present application can fully deploy the network model to the edge device even if the network model includes operators that are not supported by the NPU. Compared with limiting the use of specific operators to adapt to the NPU of the edge device, this method can ensure the completeness of the network model deployment, so that the deployed network model has a higher accuracy.
[0061] In one possible implementation, the construction of the network model is completed on a cloud server. Taking the network model as an identification model as an example, an identification model is a stacked collection of multiple operators. As a unit, an operator includes three parts: input, calculation, and output. In the network model deployment method provided in the embodiment of the present application, unsupported operators can be regarded as "section breaks", and the model is split according to their positions. And unsupported operators will not be built into sub-models, but will be transferred to CPU programming implementation. Possibly, the reason for the lack of support is that the calculation method is relatively special, and the NPU manufacturer did not plan to support this calculation method during development, or the calculation amount of the operator is too large.
[0062] exist Figure 2 Based on the content in S103, the present application embodiment also provides a possible implementation method, please refer to Figure 4 , S103 includes S103-1 and S103-2.
[0063] S103-1, when the graph complexity of the sub-model exceeds the complexity upper limit of the NPU, split the sub-model into at least two branch models.
[0064] Among them, the graph complexity of the branch model does not exceed the complexity upper limit of NPU.
[0065] It is understandable that if the graph complexity of the sub-model exceeds the complexity upper limit of the NPU, if the sub-model is directly deployed to the NPU, it may exceed the operating capacity of the NPU and affect the operation of the network model. Therefore, the sub-model can be split into branch models whose graph complexity does not exceed the complexity upper limit of the NPU. It should be noted that each branch model includes at least one operator supported by the NPU. The operators in each branch model are in a continuous state.
[0066] In a possible implementation, graph complexity includes computational complexity and / or graph structural complexity, where computational complexity is used to characterize the total computational complexity of a sub-model and graph structural complexity is used to characterize the number of minimum indivisible operators in a sub-model.
[0067] Optionally, further analyze the graph complexity of the sub-models obtained by splitting the network model, e.g. Figure 3 The graph complexity of sub-model 1 and sub-model 2 in . Graph complexity includes computational complexity and graph structural complexity. Computational complexity refers to the total computation of the sub-model, and graph structural complexity refers to the number of the smallest indivisible operators in the sub-model. Taking sub-model 2 as an example, the graph structural complexity of sub-model 2 is 9, and the computational complexity of sub-model 2 refers to the total computation of 9 operators.
[0068] In one possible implementation, complex sub-models are continuously split into two parts through hierarchical dichotomy until all models do not exceed the complexity limit, that is, the models that do not exceed the complexity limit are determined as branch models.
[0069] S103-2, deploy the branch models on the NPU respectively.
[0070] It can be understood that the sub-model is split into at least two branch models, and the split branch models are respectively deployed on corresponding NPUs, so that the NPU can run the operators in the branch models.
[0071] exist Figure 2 On the basis of how to further improve the accuracy of the network model, the embodiment of the present application also provides a possible implementation method, please refer to Figure 5 , S103 also includes S103-3 and S103-4.
[0072] S103-3, quantize each sub-model according to the parameter distribution of the network model. The parameter distribution of the network model is the parameter distribution of the network model after quantization training and convergence. In this embodiment, although the network model is split to obtain multiple sub-models, when the sub-models are quantized, the sub-models are not quantized according to the parameter distribution of the model after the split, but are quantized according to the parameter distribution of the model before the split, and the model before the split is the network model. Since each sub-model needs to be connected in series through engineering in the end, quantizing the sub-model according to the parameter distribution of the complete network model during quantization can improve the accuracy of the model after deployment.
[0073] It is understandable that when the unsplit network model is trained on the cloud server, the parameter distribution in the network model may not be applicable to the NPU when the network model converges. Therefore, it is necessary to perform quantitative analysis and transformation on the sub-model based on the parameter distribution in the converged network model to obtain a new sub-model. The parameter distribution in the new sub-model obtained by quantitative analysis and transformation is applicable to the NPU.
[0074] S103-4, deploy the quantized sub-model on the NPU.
[0075] It can be understood that the parameter distribution of the quantized sub-model is transmitted to the NPU, and the NPU can complete the deployment of the sub-model.
[0076] It should be noted that there is no necessary relationship between S103-1, S103-2 and S103-3, S103-4 in the embodiment of the present application, that is, only S103-1, S103-2 and S103-3, S103-4 can be executed. Figure 4 The sub-step of S103 shown in FIG. 104 may also be performed by only executing Figure 5 In one possible implementation, the sub-steps of S103 may be performed simultaneously. Figure 4 and Figure 5 Steps shown.
[0077] Specifically, first execute S103-1, and when the graph complexity of the sub-model exceeds the complexity upper limit of the NPU, split the sub-model into at least two branch models. Then execute S103-3, and quantize each branch model according to the parameter distribution of the network model. Finally, execute S103-4, and deploy the quantized branch model on the NPU.
[0078] In a possible implementation, the graph complexity of the sub-model does not exceed the complexity limit of the NPU. After the unsplit network model is trained to converge the network model, the sub-model can be quantitatively analyzed and transformed based on the parameter distribution in the converged network model to obtain a new sub-model. The new sub-model can then be deployed on the NPU.
[0079] In one possible implementation, the quantitative analysis includes two parts: the first part is the numerical range of the model parameters; the second part is the numerical range of the input and output of each node when the model infers the data. The specific process is as follows: first, the numerical range of the model parameters is counted; second, a large amount of data is used to infer the model to obtain the input and output data of each node, and its numerical range is counted; then, according to the numerical range of the statistical parameters and data, it is normalized to a lower precision and small storage data type supported by the NPU; finally, the model structure and parameters are combined and converted into a model format supported by the NPU. In this way, the model can be deployed on the NPU for execution with minimal accuracy loss.
[0080] In a possible implementation, the cloud server trains the unsplit network model until the network model converges.
[0081] It can be understood that the converged main network model has been determined through training, and all sub-models are split from the converged main network model. All reasonable training methods will eventually converge to the optimal result, but there will be a loss of accuracy after the network model is quantized and deployed after ordinary training. To solve this problem, quantization-aware training is used to minimize the loss of accuracy. Use the main network model for data inference, analyze the distribution of each operator parameter and output, and quantize all operator parameters and output distributions according to the goal of constraining the final result of the main model to the target result, thereby completing the quantitative analysis and parameter quantization. After the model is serialized, the structural description is converted according to the tools provided by the NPU supplier so that it can be loaded by a specific SDK for inference, thereby completing the model conversion.
[0082] exist Figure 2 On the basis of how to perform the process of refining the deployment of the network model on the edge device, the embodiment of the present application also provides a possible implementation method, such as Figure 6 As shown, the network model deployment method also includes S105.
[0083] S105, connecting all sub-models and the calculation processes of unsupported operators in series through engineering.
[0084] Among them, the concatenation order conforms to the sequential relationship in the network model.
[0085] In one possible implementation, the step of engineering the calculation processes of all sub-models and unsupported operators in series may be performed after completing the deployment of the sub-models and unsupported operators, that is, after the sub-models are deployed on the NPU and the unsupported operators are deployed on the CPU, the calculation processes of all sub-models and unsupported operators are engineered in series to complete the deployment of the final network model.
[0086] In a possible implementation, all deployed sets and computing processes that do not support operators are concatenated through engineering. The deployed sets can be sub-models or branch models, and the concatenation order conforms to the order relationship in the network model.
[0087] It can be understood that the network model can be an all-terrain obstacle detection model based on point cloud, an object recognition model, and other complex computing models. The order relationship represents the connection relationship between the input interface and the output interface of each operator. For example, the network model includes four operators, namely A, B, C, and D operators.
[0088] The sequential relationship is manifested as the input interface of operator A serving as the input port of the network model, the output interface of operator A being connected to the input interface of operator B and the input interface of operator C respectively, the output interface of operator B and the output interface of operator C being connected to the input interface of operator D, and the output interface of operator D serving as the output interface of the network model.
[0089] exist Figure 2 Based on the content in S101, the present application embodiment also provides a possible implementation method, please refer to Figure 7 , S101 includes S101-1 and S101-2.
[0090] S101-1, construct an operator dictionary based on the operators supported by the NPU in the edge device.
[0091] Optionally, operators supported by the NPU are determined according to an operator support description of the NPU.
[0092] It can be understood that the operator support description of the NPU includes the types, structures or names of all operators that the NPU can support. All operators that the NPU can support can be counted or filtered out through the operator support description.
[0093] Build an operator dictionary based on the operators supported by the NPU.
[0094] It can be understood that all operators supported by the NPU can be combined together to construct an operator dictionary. The operator dictionary includes all operators supported by the corresponding NPU.
[0095] S101-2, traverse the network model according to the operator dictionary, and filter out operators that are not supported by the NPU in the network model.
[0096] Optionally, the network model graph is traversed according to the operator dictionary to filter out unsupported operators.
[0097] It can be understood that in order to facilitate indexing and searching for unsupported operators, the operator dictionary stores a list of operators supported by the NPU. The operator list can be indexed by operator name and type.
[0098] See also Figure 8 , Figure 8 A network model deployment device is provided in an embodiment of the present application. Optionally, the network model deployment device is applied to the edge device described above.
[0099] The network model deployment device includes: a processing unit 201 and a deployment unit 202.
[0100] The processing unit 201 is configured to filter out operators not supported by the NPU in the network model according to operators supported by the NPU in the edge device. Optionally, the processing unit 201 may execute the above S101.
[0101] The processing unit 201 is further configured to use the unsupported operator as a demarcation point to split the network model to obtain at least two sub-models. Optionally, the processing unit 201 may execute the above S102.
[0102] The deployment unit 202 is used to deploy the sub-model on the NPU and deploy the unsupported operator on the CPU. Optionally, the deployment unit 202 can execute the above S103 and S104.
[0103] In a possible implementation, the deployment unit 202 is further configured to split the sub-model into at least two branch models when the graph complexity of the sub-model exceeds the complexity upper limit of the NPU; wherein the graph complexity of the branch model does not exceed the complexity upper limit of the NPU; and deploy the branch models on the NPUs respectively. Optionally, the deployment unit 202 may execute the above S103-1 and S103-2.
[0104] In a possible implementation, the deployment unit 202 is further configured to quantize each sub-model according to the parameter distribution of the network model; wherein the parameter distribution of the network model is the parameter distribution of the network model after quantization training and convergence; and deploy the quantized sub-model on the NPU. Optionally, the deployment unit 202 may perform the above-mentioned S103-3 and S103-4.
[0105] In a possible implementation, the processing unit 201 is further used to connect all sub-models and computing processes of unsupported operators in series through engineering, wherein the order of connection conforms to the order relationship in the network model. Optionally, the processing unit 201 may execute the above S105.
[0106] In a possible implementation, the processing unit 201 is further configured to construct an operator dictionary based on operators supported by the NPU in the edge device, and traverse the network model according to the operator dictionary to filter out operators not supported by the NPU in the network model. Optionally, the processing unit 201 may execute the above S101-1 and S101-2.
[0107] It should be noted that the network model deployment device provided in this embodiment can execute the method flow shown in the above method flow embodiment to achieve the corresponding technical effect. For the sake of brief description, for parts not mentioned in this embodiment, please refer to the corresponding content in the above embodiment.
[0108] The embodiment of the present application also provides a storage medium, which stores computer instructions and programs, which, when read and run, execute the network model deployment method of the above embodiment. The storage medium may include memory, flash memory, registers, or a combination thereof.
[0109] The following provides an edge device, which may be a portable mobile terminal, a computer, a server, etc. Figure 1 As shown, the above-mentioned network model deployment method can be implemented; specifically, the edge device includes: a processor 10, a memory 11, and a bus 12. The processor 10 can be a CPU. The memory 11 is used to store one or more programs. When the one or more programs are executed by the processor 10, the network model deployment method of the above-mentioned embodiment is executed.
[0110] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0111] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
[0112] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0113] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0114] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A network model deployment method, characterized in that: The method comprises: According to the operators supported by the NPU in the edge device, filter out the operators not supported by the NPU in the network model; Using the unsupported operator as a dividing point, splitting the network model to obtain at least two sub-models; Deploy the sub-model on the NPU, and deploy the unsupported operator on the CPU; The step of deploying the sub-model on the NPU includes: When the graph complexity of the sub-model exceeds the complexity upper limit of the NPU, splitting the sub-model into at least two branch models; The graph complexity of the branch model does not exceed the complexity upper limit of the NPU, and the graph complexity includes computational complexity and / or graph structural complexity, the computational complexity is used to characterize the total computational amount of the sub-model, and the graph structural complexity is used to characterize the number of minimum indivisible operators in the sub-model; The branch models are deployed on the NPUs respectively.
2. The network model deployment method according to claim 1, characterized in that: The step of deploying the sub-model on the NPU includes: Quantizing each of the sub-models according to the parameter distribution of the network model; wherein the parameter distribution of the network model is the parameter distribution of the network model after quantization training and convergence; The quantized sub-model is deployed on the NPU.
3. The network model deployment method according to claim 1, characterized in that: After deploying the sub-model on the NPU and deploying the unsupported operator on the CPU, the method further includes: Connect all sub-models and unsupported operator calculation processes in series through engineering; Among them, the concatenation order conforms to the sequential relationship in the network model.
4. The network model deployment method according to claim 1, characterized in that: The step of filtering out operators not supported by the NPU in the network model according to the operators supported by the NPU in the edge device includes: Build an operator dictionary based on the operators supported by the NPU in the edge device; The network model is traversed according to the operator dictionary, and operators not supported by the NPU in the network model are filtered out.
5. A network model deployment device, characterized in that: The device comprises: A processing unit, configured to filter out operators not supported by the NPU in the network model according to operators supported by the NPU in the edge device; The processing unit is further configured to split the network model using the unsupported operator as a dividing point to obtain at least two sub-models; A deployment unit, configured to deploy the sub-model on the NPU and deploy the unsupported operator on the CPU; The deploying the sub-model on the NPU includes: when the graph complexity of the sub-model exceeds the complexity upper limit of the NPU, splitting the sub-model into at least two branch models; wherein the graph complexity of the branch model does not exceed the complexity upper limit of the NPU, the graph complexity includes computational complexity and / or graph structure complexity, the computational complexity is used to characterize the total computational amount of the sub-model, and the graph structure complexity is used to characterize the number of minimum indivisible operators in the sub-model; and deploying the branch models on the NPU respectively.
6. The network model deployment device according to claim 5, characterized in that: The deployment unit is also used to quantize each of the sub-models according to the parameter distribution of the network model; wherein the parameter distribution of the network model is the parameter distribution of the network model after quantization training and convergence; and deploy the quantized sub-models on the NPU.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.
8. An edge device, characterized in that: include: A processor and a memory, the memory being used to store one or more programs; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Model operation method and device, terminal and storage medium
CN110458294A
