System, method, device, and medium for management of operators of artificial intelligence models

Abstract

A management system of operators of artificial intelligence models, a management method of operators of artificial intelligence models, an electronic device, and a non-transitory storage medium are provided. The management system of operators of artificial intelligence models includes: an operator manager module including an operator container for storing a plurality of operators, each operator of the plurality of operators having parameters, wherein the parameters at least include a type of an operation performed on the operator supported by the operator, a backend platform type supported by the operator, an operator identification of the operator; a front-end operator definition plug-in configured to be invoked in response to loading an operator with the parameters to provide the operator with the parameters from the operator container.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to a management system for operators of an artificial intelligence model, a management method for operators of an artificial intelligence model, an electronic device, and a non-temporary storage medium. Background Technology

[0002] Artificial intelligence models are typically neural network models used for reasoning and prediction, such as image reasoning models and speech reasoning models. Given an image containing an animal, the AI ​​model reasons and directly outputs a label indicating what the object in the image is, such as a dog or a cat. The neural network model can be a convolutional neural network, etc.

[0003] The computations of an artificial intelligence (AI) model can be implemented using operators in a computation graph. The computations of an AI model can be transformed into the corresponding computation graph using predefined computation graph transformation rules. A computation graph is a multi-graph structure used to represent the computational tasks and data flow processes of an AI model. An operator refers to various operations performed on tensors at different layers in the AI ​​model; for example, the convolution operation performed by the convolutional layers on the input data of the AI ​​model is a convolution operator. A tensor can be understood as a multidimensional array, which can have any number of dimensions, and different tensors can have different data types and shapes. An operator can be understood as a computation function whose input and output are tensors. The computation graph of an AI model can include numerous types of operators that perform operations on tensors, such as convolution operators, fully connected operators, pooling operators, transpose operators, Sobel operators, reshape operators, transpose operators, and so on.

[0004] An AI inference engine is a system component that applies logical rules to a knowledge base to infer new information. Since each new fact in the knowledge base can potentially trigger other rules in the inference engine, this process is repeated. The inference engine primarily operates in one of two modes: forward linking or backward linking, using specific rules or facts. Forward linking starts with known facts and then asserts new facts. Backward linking starts with a goal and works backward to determine which facts must be asserted to achieve the goal. The inference engine iterates through three consecutive steps: matching rules, selecting rules, and executing rules. This cycle can continue until no new rules can be matched.

[0005] With the continuous improvement of mobile computing power and the rapid development of deep learning, especially the increasing maturity of small network models, inference and prediction that were originally performed in the cloud can now be moved to the edge. Deploying and running artificial intelligence (AI) algorithms on the edge offers advantages such as low latency, protection of data privacy, and saving cloud resources.

[0006] Existing inference engine frameworks include ONNX Runtime, Mobile Neural Network (MNN), and TensorRT. For example, MNN is a lightweight deep AI inference engine that loads deep AI models on the edge for inference and prediction, suitable for various devices such as servers, PCs, mobile phones, and embedded systems.

[0007] All of the aforementioned inference engine frameworks support a certain degree of custom operators. Custom operators refer to operators written to perform specific operations based on the user's actual needs.

[0008] An operator management framework is still needed to efficiently manage and support custom operators. Summary of the Invention

[0009] According to one aspect of this application, a management system for operators of an artificial intelligence model is provided, comprising: an operator manager module including an operator container for storing a plurality of operators, each of the plurality of operators having parameters, wherein the parameters include at least a type of operation supported by the operator, a backend platform type supported by the operator, and an operator identifier of the operator; and a front-end operator definition plugin configured to be invoked in response to loading an operator having the parameters, to provide the operator having the parameters from the operator container.

[0010] According to another aspect of this application, a method for managing operators of an artificial intelligence model is provided, comprising: providing an operator manager module, including an operator container for storing a plurality of operators, each of the plurality of operators having parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and an operator identifier of the operator; and providing a front-end operator definition plugin configured to be invoked in response to loading an operator having the parameters, to provide the operator having the parameters from the operator container.

[0011] According to another aspect of this application, an electronic device is provided, comprising: a memory for storing instructions; and a processor for reading the instructions from the memory and executing a method according to an embodiment of this application.

[0012] According to another aspect of this application, a non-temporary storage medium is provided, on which instructions are stored, wherein, when read by a processor, the instructions cause the processor to perform a method according to an embodiment of this application. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 An example block diagram of a management system for operators of an artificial intelligence model according to an embodiment of this application is shown.

[0015] Figure 2 An example block diagram of a management system for an operator of an artificial intelligence model according to another embodiment of this application is shown.

[0016] Figure 3 A method for managing operators of an artificial intelligence model according to an embodiment of this application is shown.

[0017] Figure 4 A flowchart is shown of the process of adding operator plugins in the operator management method of the artificial intelligence model according to an embodiment of this application.

[0018] Figure 5 A schematic flowchart illustrating the process of loading a dynamic operator plugin according to an embodiment of this application is shown.

[0019] Figure 6 A flowchart illustrating the process of loading a model into inference and performing inference according to an embodiment of this application is shown.

[0020] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown.

[0021] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] Specific embodiments of this application will now be described in detail, with examples of the application illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this application as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0023] In the existing technology, although existing inference engine frameworks can support custom operators to a certain extent, some custom operators are supported by adding code directly through open source, such as using languages ​​like Python or C++ to write custom operators, while others are loaded at runtime using dynamic link libraries.

[0024] However, adding custom operators by adding code lacks flexibility and extensibility, and deployment is inconvenient. Loading dynamic libraries at runtime, on the other hand, prevents real-time addition and upgrades. This means that custom operators in existing inference engine frameworks do not have the same support for related functions such as graph transformation, compilation, and linking as defined operators. Current technologies only allow for a limited number of custom plugins and do not support the complex behaviors of custom operators, making it impossible to integrate them into the graph optimization process like built-in operators. Here, graph optimization refers to the optimization of the computation graph, including operators.

[0025] This application aims to provide a dynamic operator library and its management scheme in the context of the continuous development and expansion of models and operators in the field of artificial intelligence. This allows for the convenient and dynamic extension of custom operators after the inference engine framework code is released, while keeping some core processes unchanged. It supports newly extended custom operators, as well as new models and strategies related to these custom operators. This application uses a plug-in-based dynamic operator library to decouple operators, enabling the efficient parallel addition of new custom operators. Operations dependent on newly added operators can also be added to the program as plug-ins, facilitating the extension of operations on the computation graph, such as graph transformation, compilation, and linking.

[0026] Figure 1 An example block diagram of a management system 100 for an artificial intelligence model operator according to an embodiment of this application is shown.

[0027] like Figure 1 As shown, the operator management system 100 of the artificial intelligence model includes an operator manager module 110, which includes an operator container for storing multiple operators. Each operator has parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and the operator identifier of the operator. A front-end operator definition plugin 120 is configured to be invoked in response to loading an operator with parameters to provide an operator with parameters from the operator container.

[0028] Thus, after setting up an operator container in the operator manager module, any new custom operator can be stored, along with at least the types of operations supported by the new custom operator, the backend platform types supported by the operator, and the operator identifier parameters. This lays the foundation for future support of new models, strategies, and graph optimizations related to custom operators. The frontend operator definition plugin can help provide operators with specific parameters, enabling operations such as operator lookup and loading.

[0029] Figure 2 An example block diagram of a management system 200 for an operator of an artificial intelligence model according to another embodiment of this application is shown.

[0030] like Figure 2 As shown, the operator management system 200 of the artificial intelligence model includes an operator manager module 210, which includes an operator container for storing multiple operators. Each operator has parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and the operator identifier of the operator. A front-end operator definition plugin 220 is configured to be invoked in response to loading an operator with parameters to provide an operator with parameters from the operator container.

[0031] Here, in addition to the types of operations supported by the operator, the types of backend platforms supported by the operator, and the operator identifier, the parameters can also include extended parameters so that other parameters of the operator can be added in the future to expand its functionality.

[0032] In addition, the operator manager module 210 also has at least one of a lookup interface 211, a registration interface 212, and a loading interface 213, wherein the lookup interface 211 is configured to look up operators with parameters in the operator container based on parameters; the registration interface 212 is configured to register operators and their parameters to the operator container; and the loading interface 213 is configured to load front-end operator definition plugins from paths based on configurable paths and plugin matching rules.

[0033] Thus, by using the lookup interface 211, the registration interface 212, and the loading interface 213, it is convenient to register new custom operators with parameters in the operator container, and to use parameters to look up and provide various operators in the operator container, as well as to load front-end operator definition plugins in the inference engine framework.

[0034] The above dynamic operator plugins can have a unified interface: register(container, operation type, backend type, operator identifier), and the implementation of this interface will call the register interface of the operator manager module (PluginContainer).

[0035] In one embodiment, the management system 200 for operators of an artificial intelligence model further includes a front-end operator optimization definition plugin 230, configured to be invoked in response to front-end optimization of operators with parameters, to provide functionality for front-end optimization of operators with parameters.

[0036] The front-end operator optimization definition plugin 230 can be inserted through the following steps: in response to the registration of operators with parameters into the operator container, determine whether it is necessary to add a front-end optimization strategy (i.e., optimization definition) for operators with parameters; if it is determined that it is necessary to add a front-end optimization strategy for operators with parameters, generate the front-end operator optimization definition plugin 230; and add the front-end operator optimization definition plugin 230 to the system.

[0037] The front-end optimization strategy here refers to, for example, the graph optimization strategy for the front-end of an inference framework. Graph optimization strategies for operators typically involve performing graph transformations on operators in various computation graphs, such as operator replacement, operator splitting, operator fusion, and operator elimination. Then, the optimal computation graph is solved in the solution space of the transformed computation graph, thereby achieving the goal of optimizing the computation graph.

[0038] For new custom operators, optimization strategies can be defined based on their form and characteristics. For example, it can be fused or eliminated with certain operators, replaced by certain operators, and can be broken down into specific operators, or have several different fusion, elimination, replacement, or splitting methods. Therefore, by defining specific optimization strategies for specific operators in the front-end operator optimization definition plugin, these optimization strategies can be applied to specific operators during subsequent graph optimization, followed by overall graph optimization.

[0039] Thus, since new custom operators may be graph optimized together with other operators in the computation graph, according to the embodiments of this application, when a new custom operator is registered into an operator container, a front-end operator optimization definition plugin related to the custom operator can be added synchronously to support the front-end optimization strategy of the custom operator.

[0040] In one embodiment, the management system 200 for operators of an artificial intelligence model further includes: an operator transformation plugin 240 configured to be invoked in response to transforming an operator with parameters, to provide the functionality of transforming an operator with parameters to perform graph optimization of a computation graph including the operator, wherein the transformation includes one or more of operator fusion, operator splitting, operator replacement, and operator elimination.

[0041] Operator transformation plugin 240 is inserted through the following steps: in response to an operator with parameters being registered in the operator container, it is determined whether a transformation strategy needs to be added for the operator with parameters; if it is determined that a transformation strategy needs to be added for the operator with parameters, operator transformation plugin 240 is generated; operator transformation plugin 240 is added to the system.

[0042] Operator substitution replaces an unsupported operator with a combination of one or more supported operators. The backend can include, for example, a Graphics Processing Unit (GPU), a Central Processing Unit (CPU), or a Field-Programmable Array (FPGA). Currently, computation graph execution in various deep learning frameworks typically occurs on a single computing device. Frameworks like TensorFlow and PyTorch achieve excellent results on GPU backends but do not support other backend devices such as FPGAs. Therefore, it is necessary to replace unsupported operators with combinations of supported operators. Operator substitution can also include operator type conversion. Operator type conversion converts an operator from one type to another.

[0043] Operator splitting involves dividing an operator with a computational cost exceeding a first predetermined threshold into multiple operators of the same type with a computational cost less than a second predetermined threshold, where the first predetermined threshold is greater than or equal to the second predetermined threshold. In other words, it splits a large, computationally intensive operator into several smaller, less computationally intensive operators. Specifically, a large, computationally intensive operator can be split into several smaller, less computationally intensive operators of the same type. For example, a large Conv operator can be split into multiple smaller Conv operators sharing the same input, collectively implementing the functionality of the large Conv operator. Alternatively, a large, computationally intensive operator of a type not supported by the backend can be split into smaller, less computationally intensive operators of different types supported by the backend.

[0044] Operator fusion is an operation that aggregates multiple operators that meet a certain condition into a single operator. For example, the condition might be: if multiple Conv operators share the same input, these Conv operators can be merged into a larger Conv operator. Alternatively, the condition could be to merge Conv operators, batch normalization (BN) operators, and ReLU operators into a new operator, such as the CBR operator, to reduce operator scheduling overhead and data access overhead between operators. For instance, suppose we want to compute ReLU(X+Y), where X and Y are both of length L. In TensorFlow, this corresponds to two operators: Add and ReLU. When performing Add computation, X and Y are first read from memory, and then the computation result is written back to memory; therefore, Add computation requires reading and writing 3L of data from memory. Similarly, ReLU computation requires reading the input data from memory and then writing the result back to memory; therefore, ReLU computation requires reading and writing 2L of data from memory. The total amount of data read and written by Add and ReLU combined is 5L. If we combine them into a single operator, read X and Y from memory, perform addition first, then ReLU calculation, and finally save the final result back into memory, the total amount of data read and written will only be 3L.

[0045] Operator elimination is an operation that eliminates operators that meet a certain condition. For example, if two operators are transpose(m, n) and the following transpose(n, m), it means that the operation of the two operators does not actually change the tensor. Therefore, transpose(m, n) and the following transpose(n, m) can be eliminated directly (where m and n are positive integers).

[0046] Graph optimization can reduce the number of operators, for example, through operator fusion or elimination, or by performing operator fusion or elimination after operator replacement or splitting, because for the framework, moving from one operator to another means data transfer. For example, fusing Batch Normalization (BN) operators with Conv operators can reduce the amount of data transferred from the original Conv operator output to the BN operator input. Graph optimization can also accommodate hardware limitations, such as replacing operators not supported by the backend hardware with those that are supported, for example, replacing various operators with convolution operators.

[0047] Thus, when inserting a custom operator, an operator transformation plugin can be inserted to support the application of various transformation operations to the computation graph with the inserted custom operator. This is used to perform graph optimization by finding the optimal solution of graph optimization of the computation graph after inserting the custom operator, thereby providing support for graph transformation and graph optimization of custom operators in the inference engine framework.

[0048] In this way, an operator transformation plugin can be added to the inference engine framework. Based on the specific optimization strategies of specific operators supported by the front-end operator optimization definition plugin and the existing optimization strategies of existing operators, the operator transformation plugin can be used to transform the operators in the computation graph, which is more conducive to supporting specific graph optimization of computation graphs with newly added custom operators.

[0049] After graph optimization, the newly added custom operators may be converted into new transformation operators that are different from the original custom operators. Of course, they may also remain unchanged after graph optimization. Therefore, custom operators after graph optimization are collectively referred to as operators with parameters or new transformation operators in the optimized computation graph.

[0050] In one embodiment, the management system 200 for operators of an artificial intelligence model may further include an operator compilation plugin 250 configured to be invoked in response to compiling a parameterized operator or a new transformation operator in an optimized computation graph, to provide the ability to compile a parameterized operator or a new transformation operator in an optimized computation graph.

[0051] Operator compiler plugin 250 is inserted through the following steps: in response to an operator with parameters being registered in the operator container, determining whether the compiler plugin set is capable of compiling the operator with parameters or the new transformation operator in the optimized computation graph; if it is determined that the compiler plugin set is not capable of compiling the operator with parameters or the new transformation operator in the optimized computation graph, then generating operator compiler plugin 250; and adding operator compiler plugin 250 to the system (e.g., to the compiler plugin set).

[0052] Operator Compiler Plugin 250 can compile operators with parameters or new transformation operators in an optimized computation graph for subsequent linking into executable units.

[0053] Compilation typically begins by generating an intermediate representation. The specific process will not be detailed here due to space limitations.

[0054] Here, the compilation plugin set can be an existing set of plugins in the inference engine framework for compiling operators. If it is determined that the existing compilation plugin set cannot compile a new operator with parameters or a new transformation operator in the optimized computation graph, it means that the new operator (new transformation operator) lacks a corresponding compilation plugin. Therefore, a configuration for compiling the backend code of the new operator (new transformation operator) to implement the new operator (new transformation operator) and an operator compilation plugin 250 for generating intermediate expressions are generated.

[0055] Thus, an operator compilation plugin can be added to the inference engine framework to compile operators with parameters or new transformation operators in the optimized computation graph.

[0056] In one embodiment, the management system 200 for operators of an artificial intelligence model may further include: an operator linking plugin 260 configured to be invoked in response to linking operators with parameters or new transformation operators in an optimized computation graph, to provide the functionality of linking operators with parameters or new transformation operators in an optimized computation graph.

[0057] Operator linker 260 is inserted through the following steps: in response to an operator with parameters being registered in the operator container, determining whether the linker set can link the operator with parameters or the new transformation operator in the optimized computation graph; if it is determined that the linker set cannot link the operator with parameters or the new transformation operator in the optimized computation graph, then operator linker 260 is generated; operator compilation plugin 260 is added to the system (e.g., to the linker set).

[0058] Operator Linker 260 can link the compilation results of operators with parameters or new transformation operators in an optimized computation graph to generate executable units.

[0059] Here, the linker set can be an existing set of linkers in the inference engine framework. If it is determined that the existing linker set cannot link a new operator with parameters or a new transformation operator in the optimized computation graph, it means that the new operator (new transformation operator) lacks a linker for it. Therefore, an operator linker 250 is generated to link the new operator (new transformation operator) to correspond to the executable binary code generated by the backend implementation based on the intermediate expression obtained from the previous compilation.

[0060] Thus, an operator linking plugin can be added to the inference engine framework to link the compilation results of operators with parameters or new transformation operators in the optimized computation graph to generate an executable unit for subsequent execution.

[0061] In one embodiment, the management system 200 for operators of an artificial intelligence model may further include: an operator execution plugin 270 configured to be invoked in response to the execution of an operator with parameters, to provide the functionality to execute an operator with parameters.

[0062] Operator execution plugin 270 is inserted through the following steps: in response to an operator with parameters being registered in the operator container, determining whether the operator execution set can execute the operator with parameters or the new transformation operator in the optimized computation graph; if it is determined that the operator execution set cannot execute the operator with parameters or the new transformation operator in the optimized computation graph, then generating operator execution plugin 270; and adding operator execution plugin 270 to the system (e.g., to the operator execution set).

[0063] Thus, an operator execution plugin can be added to the inference engine framework to call the backend to execute the previously generated executable unit (executable binary code).

[0064] Here, the operator execution set can be an existing set of plugins in the inference engine framework for executing operators. If it is determined that the existing operator execution set cannot link to a new operator with parameters or a new transformation operator in the optimized computation graph, it means that the new operator (new transformation operator) lacks an execution plugin for it. Therefore, an operator execution plugin 250 for executing the new operator (new transformation operator) is generated.

[0065] Let's take a concrete example to illustrate the process after adding a new backend operator, conv.

[0066] Assuming the backend of the new operator directly uses the conv operator and does not need to be replaced, the following process can be performed:

[0067] a. Add a front-end conv operator.

[0068] b. If the front-end directly uses the conv operator and does not require an optimization strategy, then no front-end optimization operator will be added.

[0069] c. If the backend directly uses the conv operator without replacement operations, then no backend transformation operator needs to be added.

[0070] d. Add a compilation plugin for the backend conv (for example, for the CPU backend, implement the configuration of CPU code for conv operators and generate intermediate expressions within the newly added plugin).

[0071] e. Add a linker plugin for the backend conv (e.g., for the CPU backend, implement the generation of executable binary code based on the intermediate expression of conv within the newly added plugin).

[0072] f. Add an execution plugin for the conv operator (e.g., for the CPU backend, call the execution method of the CPU backend within the newly added plugin).

[0073] Below, we will use a specific example to illustrate the process after adding a new operator, conv, that is not supported by the backend.

[0074] Add a conv operator, assuming the backend wants to use CBAR(conv+bias+Add+Relu).

[0075] a. Add a front-end conv operator.

[0076] b. The front-end strategy decides to perform CB fusion and add a front-end optimization plugin (to achieve fusion of conv+bias dual operators and obtain a new front-end CB operator).

[0077] c. If the backend strategy decides to perform CBAR fusion, then a CB operator conversion plugin is added (which realizes the fusion of the frontend CB+Add+ReLU operators to obtain a new backend CBAR operator).

[0078] d. Add a CBAR compilation plugin for the backend (e.g., for the CPU backend, implement the configuration of the CPU code for the CBAR operator and generate intermediate expressions within the newly added plugin).

[0079] e. Add a linking plugin for the backend CBAR (for example, for the CPU backend, implement the generation of executable binary code based on the intermediate expression of CBAR within the newly added plugin).

[0080] Add an execution plugin for the CBAR operator (for example, for the CPU backend, call the execution method of the CPU backend within the newly added plugin).

[0081] In this way, by using a dynamic operator library, the inference engine only needs to load the necessary operators and plugins at runtime, reducing the program's runtime size and increasing efficiency. After product release, upgrades can be easily achieved by releasing plugins for new operators, thus supporting hot deployment. This provides flexible expansion capabilities as models and operators in the field of artificial intelligence continue to evolve and expand.

[0082] Figure 3 A method 300 for managing operators of an artificial intelligence model according to an embodiment of this application is shown.

[0083] like Figure 3 As shown, the operator management method 300 for the artificial intelligence model includes: step 310, providing an operator manager module, including an operator container for storing multiple operators, each of the multiple operators having parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and the operator identifier of the operator; step 320, providing a front-end operator definition plugin, configured to be invoked in response to loading an operator with parameters, to provide an operator with parameters from the operator container.

[0084] In one embodiment, the operator management method 300 for an artificial intelligence model includes: further comprising: providing at least one of a search interface, a registration interface, and a loading interface for an operator manager module, wherein the search interface is configured to search for operators with parameters in an operator container based on parameters; the registration interface is configured to register operators and their parameters to the operator container; and the loading interface is configured to load front-end operator definition plugins from a path based on configurable path and plugin matching rules.

[0085] The above dynamic operator plugins can have a unified interface: register(container, operation type, backend type, operator identifier), and the implementation of this interface will call the register interface of the operator manager module (PluginContainer).

[0086] Figure 4 A flowchart is shown of the process of adding operator plugins in the operator management method 400 of the artificial intelligence model according to an embodiment of this application.

[0087] In one embodiment, when it is determined that a front-end operator needs to be added for a new operator (step 401), the operator management method 400 of the artificial intelligence model includes: providing a front-end operator definition plugin (step 402), configured to be invoked in response to loading an operator with parameters, to provide an operator with parameters from an operator container.

[0088] In one embodiment, the method 300 for managing operators of an artificial intelligence model further includes: providing a front-end operator optimization definition plugin, configured to be invoked in response to front-end optimization of operators with parameters, to provide functionality for front-end optimization of operators with parameters.

[0089] Providing a front-end operator optimization definition plugin includes: in response to an operator with parameters being registered in the operator container, determining whether a front-end optimization strategy needs to be added for the operator with parameters (step 403); if it is determined that a front-end optimization strategy needs to be added for the operator with parameters, generating a front-end operator optimization definition plugin; and adding the front-end operator optimization definition plugin to the system (step 404).

[0090] In one embodiment, the method 300 for managing operators of an artificial intelligence model further includes: providing an operator transformation plugin configured to be invoked in response to transforming an operator with parameters, to provide functionality for transforming an operator with parameters to perform graph optimization of a computation graph including the operator, wherein the transformation includes one or more of operator fusion, operator splitting, operator replacement, and operator elimination.

[0091] Providing an operator transformation plugin includes: in response to an operator with parameters being registered in an operator container, determining whether a transformation strategy needs to be added for the operator with parameters (step 405); if it is determined that a transformation strategy needs to be added for the operator with parameters, generating an operator transformation plugin; and adding the operator transformation plugin to the system (step 406).

[0092] In one embodiment, the method 300 for managing operators of an artificial intelligence model further includes: providing an operator compilation plugin configured to be invoked in response to compiling a parameterized operator or a new transformation operator in an optimized computation graph, to provide the ability to compile a parameterized operator or a new transformation operator in an optimized computation graph.

[0093] Providing operator compilation plugins includes: in response to an operator with parameters being registered in an operator container, determining whether the compilation plugin set can compile the operator with parameters or the new transformation operator in the optimized computation graph (step 407); if it is determined that the compilation plugin set cannot compile the operator with parameters or the new transformation operator in the optimized computation graph, then generating an operator compilation plugin; and adding the operator compilation plugin to the system (step 408).

[0094] In one embodiment, the method 300 for managing operators of an artificial intelligence model further includes: providing an operator linking plugin configured to be invoked in response to linking operators with parameters or new transformation operators in an optimized computation graph, to provide the functionality to link operators with parameters or new transformation operators in an optimized computation graph.

[0095] Providing operator linking plugins includes: in response to an operator with parameters being registered in an operator container, determining whether the set of linking plugins can link the operator with parameters or the new transformation operator in the optimized computation graph (step 409); if it is determined that the set of linking plugins cannot link the operator with parameters or the new transformation operator in the optimized computation graph, then generating an operator linking plugin; and adding the operator compilation plugin to the system (step 410).

[0096] In one embodiment, the method 300 for managing operators of an artificial intelligence model further includes: providing an operator execution plugin configured to be invoked in response to execution of an operator with parameters, to provide the functionality to execute an operator with parameters.

[0097] Providing an operator compilation plugin includes: in response to an operator with parameters being registered in an operator container, determining whether the operator execution set can execute the operator with parameters or the new transformation operator in the optimized computation graph (step 411); if it is determined that the operator execution set cannot execute the operator with parameters or the new transformation operator in the optimized computation graph, then generating an operator execution plugin; and adding the operator compilation plugin to the system (step 412).

[0098] In this way, by using a dynamic operator library, the inference engine only needs to load the necessary operators and plugins at runtime, reducing the program's runtime size and increasing efficiency. After product release, upgrades can be easily achieved by releasing plugins for new operators, thus supporting hot deployment. This provides flexible expansion capabilities as models and operators in the field of artificial intelligence continue to evolve and expand.

[0099] Figure 5 A schematic flowchart illustrating the process of loading a dynamic operator plugin according to an embodiment of this application is shown.

[0100] like Figure 5 As shown, when a dynamic operator plugin needs to be loaded, in step 510, the dynamic operator plugin is searched in a predetermined directory or a specified directory. In step 520, it is confirmed whether the dynamic operator plugin has been found. If yes, in step 530, it is confirmed whether the loading was successful; otherwise, the process terminates. If yes, in step 540, plugin.register(pluginContainer) is called to run the dynamic operator plugin; otherwise, the process terminates.

[0101] Figure 6 A flowchart illustrating the process of loading a model into inference and performing inference according to an embodiment of this application is shown.

[0102] like Figure 6 As shown, each step of the operation on the operator obtains the necessary plugins from the dynamic operator management container, including operator expression, front-end optimization, operator transformation, operator compilation, operator linking, and operator execution.

[0103] Specifically, during model loading, the model parser obtains front-end operator definition plugins from the dynamic operator library through the dynamic operator loading, management, and lookup interface in order to load the model.

[0104] During front-end optimization, the front-end optimizer obtains front-end operator optimization definition plugins from the dynamic operator library through the dynamic operator loading, management, and search interface in order to perform front-end optimization.

[0105] During graph transformation (optimization), the graph converter obtains operator transformation plugins from the dynamic operator library through the dynamic operator loading, management, and search interface in order to perform graph transformation (optimization).

[0106] During operator compilation, the compilation interface obtains operator compilation plugins from the dynamic operator library through the dynamic operator loading, management, and search interface in order to perform operator compilation.

[0107] During operator linking, the linking interface obtains operator (subgraph) linking plugins from the dynamic operator library through the dynamic operator loading, management, and search interface in order to perform operator linking.

[0108] During operator execution, the operator execution interface obtains the operator execution plugin from the dynamic operator library through the dynamic operator loading, management and search interface in order to perform operator execution.

[0109] In this way, by using a dynamic operator library, the inference engine only needs to load the necessary operators and plugins at runtime, reducing the program's runtime size and increasing efficiency. After product release, upgrades can be easily achieved by releasing plugins for new operators, thus supporting hot deployment. This provides flexible expansion capabilities as models and operators in the field of artificial intelligence continue to evolve and expand.

[0110] Figure 7 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present application is shown.

[0111] The electronic device may include a processor (H1); and a storage medium (H2) coupled to the processor (H1) and storing computer-executable instructions therein for performing the steps of various methods of the embodiments of this application when executed by the processor.

[0112] The processor (H1) may include, but is not limited to, one or more processors or microprocessors.

[0113] Storage media (H2) may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0114] In addition, the electronic device may also include a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (e.g., keyboard, mouse, speaker, etc.).

[0115] The processor (H1) can communicate with external devices (H5, H6, etc.) via the I / O bus (H4) through a wired or wireless network (not shown).

[0116] The storage medium (H2) may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described in this technology when executed by the processor (H1).

[0117] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0118] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of the present disclosure is shown.

[0119] like Figure 8 As shown, the computer-readable storage medium 820 stores instructions, such as computer-readable instruction 810. When the computer-readable instruction 810 is executed by a processor, the various methods described above can be performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the computer-readable storage medium 820 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instruction 810 stored on the computer-readable storage medium 820, the various methods described above can be performed.

[0120] Of course, the specific embodiments described above are merely examples and not limitations. Those skilled in the art can integrate and combine some steps and devices from the various embodiments described separately above to achieve the effects of this application based on the concept of this application. Such integrated and combined embodiments are also included in this application, but will not be described one by one here.

[0121] Note that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations; these details do not restrict this application from being implemented using the aforementioned specific details.

[0122] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0123] The flowcharts and method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as "then," "next," etc., are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to a singular element, such as the use of the articles "a," "one," or "the," is not to be construed as limiting that element to the singular.

[0124] Furthermore, the steps and apparatus in the various embodiments herein are not limited to any one embodiment. In fact, new embodiments can be conceived by combining relevant steps and apparatus in the various embodiments herein based on the concepts of this application, and these new embodiments are also included within the scope of this application.

[0125] The various operations of the methods described above can be performed by any suitable means capable of performing the corresponding functions. Such means may include various hardware and / or software components and / or modules, including but not limited to hardware circuits, application-specific integrated circuits (ASICs), or processors.

[0126] The various exemplified logic blocks, modules, and circuits described herein can be implemented or performed using a general-purpose processor, digital signal processor (DSP), ASIC, field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, but alternatively, it can be any commercially available processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a microprocessor cooperating with a DSP core, or any other such configuration.

[0127] The steps of the methods or algorithms described in this disclosure can be directly embedded in hardware, in a software module executed by a processor, or a combination of both. The software module can reside in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. The storage medium can be coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integral with the processor. The software module can be a single instruction or many instructions, and can be distributed across several different code segments, different programs, and across multiple storage media.

[0128] The methods disclosed herein include actions for implementing the described methods. The methods and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims.

[0129] The above functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital universal disc (DVD), floppy disk, and Blu-ray disc, wherein a disc typically magnetically reproduces data, while a disc optically reproduces data using lasers.

[0130] Therefore, a computer program product can perform the operations given herein. For example, such a computer program product can be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0131] Software or instructions can also be transmitted via a transmission medium. For example, software can be transmitted from a website, server, or other remote source using transmission media such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave.

[0132] Furthermore, modules and / or other suitable means for carrying out the methods and techniques described herein can be downloaded and / or obtained by user terminals and / or base stations as appropriate. For example, such a device can be coupled to a server to facilitate the transmission of means for carrying out the methods described herein. Alternatively, the various methods described herein can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that user terminals and / or base stations can obtain the various methods when coupled to the device or when storage components are provided to the device. Furthermore, any other suitable techniques for providing the methods and techniques described herein to the device can be utilized.

[0133] Other examples and implementations are within the scope and spirit of this disclosure and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Features implementing the functions can also be physically located in various places, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the described examples are preferred or better than other examples.

[0134] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0135] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0136] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A management system for operators of an artificial intelligence model, comprising: The operator manager module includes an operator container for storing multiple operators. Each operator has parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and the operator identifier of the operator. A front-end operator definition plugin is configured to be invoked in response to loading an operator having the parameters, in order to provide the operator having the parameters from the operator container; A front-end operator optimization definition plugin is configured to be invoked in response to front-end optimization of operators having the parameters, thereby providing the functionality to optimize operators having the parameters. The front-end operator optimization definition plugin is inserted through the following steps: In response to an operator having the parameters being registered in the operator container, it is determined whether a front-end optimization strategy needs to be added for the operator having the parameters; If it is determined that a front-end optimization strategy needs to be added for operators with the parameters, then the front-end operator optimization definition plugin is generated. Add the front-end operator optimization definition plugin to the system.

2. The system according to claim 1, wherein, The operator manager module also has at least one of a lookup interface, a registration interface, and a loading interface. The lookup interface is configured to search for an operator with the parameters in the operator container based on the parameters. The registration interface is configured to register operators and their parameters to the operator container; The loading interface is configured to load the front-end operator definition plugin from the path based on configurable path and plugin matching rules.

3. The system according to claim 1, further comprising: An operator transformation plugin is configured to be invoked in response to transforming an operator having the parameters, to provide functionality for transforming the operator having the parameters to perform graph optimization of the computation graph including the operator, wherein the transformation includes one or more of operator fusion, operator splitting, operator replacement, and operator elimination. The operator transformation plugin is inserted through the following steps: In response to an operator having the parameters being registered in the operator container, it is determined whether a transformation strategy needs to be added for the operator having the parameters; If it is determined that a transformation strategy needs to be added for operators with the aforementioned parameters, then the operator transformation plugin is generated. Add the operator transformation plugin to the system.

4. The system according to claim 3, further comprising: An operator compilation plugin, configured to be invoked in response to compiling an operator having the stated parameters or a new transformation operator in an optimized computation graph, provides the functionality to compile operators having the stated parameters or new transformation operators in an optimized computation graph. The operator compilation plugin is inserted through the following steps: In response to an operator having the parameters being registered in the operator container, it is determined whether the compilation plugin set is capable of compiling the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the compilation plugin set cannot compile operators with the parameters or new transformation operators in the optimized computation graph, then the operator compilation plugin is generated. Add the operator compilation plugin to the system.

5. The system according to claim 4, further comprising: An operator linking plugin, configured to be invoked in response to linking operators with the stated parameters or new transformation operators in an optimized computation graph, provides the functionality to link operators with the stated parameters or new transformation operators in an optimized computation graph. The operator linking plugin is inserted through the following steps: In response to an operator having the parameters being registered in the operator container, it is determined whether the linker set can link the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the set of linking plugins cannot link operators with the parameters or new transformation operators in the optimized computation graph, then the operator linking plugin is generated. Add the operator linking plugin to the system.

6. The system according to claim 5, further comprising: An operator execution plugin, configured to be invoked in response to execution of an operator having the parameters, provides the functionality to execute operators having the parameters. The operator execution plugin is inserted through the following steps: In response to an operator having the parameters being registered in the operator container, it is determined whether the operator execution set is capable of executing the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the operator execution set cannot execute the operator with the parameters or the new transformation operator in the optimized computation graph, then the operator execution plugin is generated; Add the operator execution plugin to the system.

7. A method for managing operators in an artificial intelligence model, comprising: An operator manager module is provided, including an operator container for storing multiple operators. Each of the multiple operators has parameters, wherein the parameters include at least the type of operation supported by the operator, the type of backend platform supported by the operator, and the operator identifier of the operator. A front-end operator definition plugin is provided, configured to be invoked in response to loading an operator having the parameters, to provide the operator having the parameters from the operator container; A front-end operator optimization definition plugin is provided, configured to be invoked in response to front-end optimization of operators having the parameters, to provide the functionality of front-end optimization of operators having the parameters. The plugin that provides front-end operator optimization definitions includes: In response to an operator having the parameters being registered in the operator container, it is determined whether a front-end optimization strategy needs to be added for the operator having the parameters; If it is determined that a front-end optimization strategy needs to be added for operators with the parameters, then the front-end operator optimization definition plugin is generated. Add the aforementioned front-end operator optimization definition plugin to the operator management system of the artificial intelligence model.

8. The method according to claim 7, further comprising: Provide at least one of the search interface, registration interface, and loading interface of the operator manager module. The lookup interface is configured to search for an operator with the parameters in the operator container based on the parameters. The registration interface is configured to register operators and their parameters to the operator container; The loading interface is configured to load the front-end operator definition plugin from the path based on configurable path and plugin matching rules.

9. The method according to claim 7, further comprising: An operator transformation plugin is provided, configured to be invoked in response to transforming an operator having the parameters, to provide functionality for transforming the operator having the parameters to perform graph optimization of the computation graph including the operator, wherein the transformation includes one or more of operator fusion, operator splitting, operator replacement, and operator elimination. The operator transformation plugin includes: In response to an operator having the parameters being registered in the operator container, it is determined whether a transformation strategy needs to be added for the operator having the parameters; If it is determined that a transformation strategy needs to be added for operators with the aforementioned parameters, then the operator transformation plugin is generated. Add the operator transformation plugin to the system.

10. The method of claim 9, further comprising: An operator compilation plugin is provided, configured to be invoked in response to compiling an operator having the stated parameters or a new transformation operator in an optimized computation graph, to provide the functionality to compile an operator having the stated parameters or a new transformation operator in an optimized computation graph. The operator compilation plugin includes: In response to an operator having the parameters being registered in the operator container, it is determined whether the compilation plugin set is capable of compiling the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the compilation plugin set cannot compile operators with the parameters or new transformation operators in the optimized computation graph, then the operator compilation plugin is generated. Add the operator compilation plugin to the system.

11. The method of claim 10, further comprising: An operator linking plugin is provided, configured to be invoked in response to linking operators with the stated parameters or new transformation operators in the optimized computation graph, to provide the functionality of linking operators with the stated parameters or new transformation operators in the optimized computation graph. The operator linking plugin includes: In response to an operator having the parameters being registered in the operator container, it is determined whether the linker set can link the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the set of linking plugins cannot link operators with the parameters or new transformation operators in the optimized computation graph, then the operator linking plugin is generated. Add the operator linking plugin to the system.

12. The method of claim 11, further comprising: An operator execution plugin is provided, configured to be invoked in response to execution of an operator having the parameters, to provide the functionality to execute operators having the parameters. The operator execution plugin includes: In response to an operator having the parameters being registered in the operator container, it is determined whether the operator execution set is capable of executing the operator having the parameters or the new transformation operator in the optimized computation graph; If it is determined that the operator execution set cannot execute the operator with the parameters or the new transformation operator in the optimized computation graph, then the operator execution plugin is generated; Add the operator execution plugin to the system.

13. An electronic device, comprising: Memory, used to store instructions; A processor for reading instructions from the memory and executing the method as described in any one of claims 7-12.

14. A non-temporary storage medium having instructions stored thereon, in, When the instruction is read by the processor, it causes the processor to perform the method as described in any one of claims 7-12.

Images