Operator fusion method and system of reasoning engine, electronic device and medium

Abstract

Provided are an operator fusion method, system, electronic device, and non-transitory storage medium of an inference engine of a deep learning model, the method comprising: obtaining a configuration file, the configuration file comprising at least the types of multiple operators to be fused in a user-defined operator fusion strategy and the type of a fused operator; and fusing each operator in a network graph of a deep learning model according to the configuration file to obtain a fused network graph.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to operator fusion methods, systems, electronic devices, and non-temporary storage media for inference engines of deep learning models. Background Technology

[0002] Artificial intelligence models are typically neural network models used for reasoning and prediction, such as image reasoning models and speech reasoning models. Given an image containing an animal, the AI ​​model reasons and directly outputs a label indicating what the object in the image is, such as a dog or a cat. The neural network model can be a convolutional neural network, etc.

[0003] The computations of an artificial intelligence (AI) model can be implemented using operators in a computation graph. The computations of an AI model can be transformed into the corresponding computation graph using predefined computation graph transformation rules. A computation graph is a multi-graph structure used to represent the computational tasks and data flow processes of an AI model. An operator refers to various operations performed on tensors at different layers in the AI ​​model; for example, the convolution operation performed by the convolutional layers on the input data of the AI ​​model is a convolution operator. A tensor can be understood as a multidimensional array, which can have any number of dimensions, and different tensors can have different data types and shapes. An operator can be understood as a computation function whose input and output are tensors. The computation graph of an AI model can include numerous types of operators that perform operations on tensors, such as convolution operators, fully connected operators, pooling operators, transpose operators, Sobel operators, reshape operators, transpose operators, and so on.

[0004] An AI inference engine is a system component that applies logical rules to a knowledge base to infer new information. Since each new fact in the knowledge base can potentially trigger other rules in the inference engine, this process is repeated. The inference engine primarily operates in one of two modes: forward linking or backward linking, using specific rules or facts. Forward linking starts with known facts and then asserts new facts. Backward linking starts with a goal and works backward to determine which facts must be asserted to achieve the goal. The inference engine iterates through three consecutive steps: matching rules, selecting rules, and executing rules. This cycle can continue until no new rules can be matched.

[0005] Existing inference engine frameworks include ONNX Runtime, Mobile Neural Network (MNN), and TensorRT. For example, MNN is a lightweight deep AI inference engine that loads deep AI models on the edge for inference and prediction, suitable for various devices such as servers, PCs, mobile phones, and embedded systems.

[0006] All of the aforementioned inference engine frameworks support operator fusion, which is a key factor in improving the performance of inference engines. It can merge two or more operators into one operator, thereby reducing the number of network layers, achieving effects such as reducing memory access overhead, power consumption, computational load, and kernel startup time.

[0007] A solution is needed to improve operator fusion performance within the inference engine framework. Summary of the Invention

[0008] According to one aspect of this application, an operator fusion method for an inference engine of a deep learning model is provided, comprising: obtaining a configuration file, the configuration file including at least the types of multiple operators to be fused in a user-defined operator fusion strategy and the types of the fused operators; and fusing the various operators in the network graph of the deep learning model according to the configuration file to obtain the fused network graph.

[0009] According to another aspect of this application, an operator fusion system for an inference engine of a deep learning model is provided, comprising: a configuration device configured to obtain a configuration file, the configuration file including at least the types of multiple operators to be fused in a user-defined operator fusion strategy and the types of the fused operators; and a fusion device configured to fuse the various operators in the network graph of the deep learning model according to the configuration file to obtain the fused network graph.

[0010] According to another aspect of this application, an electronic device is provided, comprising: a memory for storing instructions; and a processor for reading the instructions from the memory and executing the method described according to embodiments of this application.

[0011] According to another aspect of this application, a non-temporary storage medium is provided, on which instructions are stored, wherein, when read by a processor, the instructions cause the processor to perform the method described according to an embodiment of this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 The diagram illustrates the operator fusion of existing technologies.

[0014] Figure 2 A flowchart illustrating an operator fusion method for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0015] Figure 3A A schematic diagram illustrating a first example of fusing multiple operators of the types of the multiple operators to be fused into an operator of the type of the fused operator according to an embodiment of the present application, based on a configuration file.

[0016] Figure 3B A schematic diagram illustrating a second example of fusing multiple operators of the types of the multiple operators to be fused into an operator of the type of the fused operator according to an embodiment of the present application, based on a configuration file.

[0017] Figure 4 A flowchart illustrating an operator fusion method for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0018] Figure 5 A block diagram of an operator fusion system for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0019] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of this application is shown.

[0020] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of this application is shown. Detailed Implementation

[0021] Specific embodiments of this application will now be described in detail, with examples of the application illustrated in the accompanying drawings. Although this application will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit this application to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of this application as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0022] Operator fusion is an operation that combines multiple operators that meet a certain condition into a single operator. Figure 1 The diagram illustrates operator fusion in the prior art. For example... Figure 1 As shown above, for example, there are three operators: the input of operator OpLeft is input X, the input of operator OpRight is input Y, and the outputs of operator OpLeft and operator OpRight are passed through operator OpDown to obtain the output.

[0023] For such operators, fusion is performed, such as Figure 1 The three operators Op1, Op2, and Op3 on the left are merged into Figure 1 An operator on the right, Op22.

[0024] For example, the condition is that if multiple Conv operators share the same input, these Conv operators can be merged into a larger Conv operator. Alternatively, the condition is to merge Conv operators, batch normalization (BN) operators, and ReLU operators into a new operator, such as the CBR operator, to reduce operator scheduling overhead and data access overhead between operators. For instance, suppose we want to compute ReLU(X+Y), where X and Y are both of length L. In a tensor stream, this corresponds to two operators: Add and ReLU. When performing Add computation, X and Y are first read from memory, and then the computation result is written back to memory; therefore, Add computation requires reading and writing 3L of data from memory. Similarly, ReLU computation requires reading the input data from memory and then writing the result back to memory; therefore, ReLU computation requires reading and writing 2L of data from memory. The total amount of data read and written by Add and ReLU combined is 5L. If we combine them into a single operator, read X and Y from memory, perform addition first, then ReLU calculation, and finally save the final result back into memory, the total amount of data read and written will only be 3L.

[0025] The strategies that fuse multiple types of operators into a single type are called operator fusion strategies. Existing operator fusion techniques employ one or more fixed strategies; however, these strategies are often not universally applicable to all models. Current inference engines mostly use fixed operator fusion strategies, ignoring the specificities between various AI models with different operators, resulting in low compatibility between the operator fusion strategy and the AI ​​model. Inference models within the same framework often incur redundant computations after conversion, which the inference engine struggles to identify. The operator fusion strategies used often lead to low fusion levels in the AI ​​models, limiting further performance improvements. Existing inference engine fusion techniques rarely allow for user customization of fusion strategies. There is a lack of fusion strategies for user-defined operators or operators that the inference engine cannot recognize. Existing inference engine fusion strategies are opaque to users, making it difficult for them to know the specific structure of the fused graph. Using fixed fusion strategies for multiple models, while considering the adaptability of multiple models, limits the flexibility and scope of the fusion strategy. The lack of reverting in fusion prevents numerous small operators from being merged, significantly limiting the performance improvement of the inference engine. The operator fusion strategy of the inference engine is opaque to the user, making it difficult for the user's AI model to match existing operator fusion strategies. The operator fusion strategy of the inference engine is not user-controlled; even seemingly excellent fusion strategies are sometimes difficult to implement. The fusion strategy is invisible to the user, and the merged network graph is not readily available, making it difficult to pinpoint problems with the fusion strategy and key performance improvement points. The merged operators rely on backend operator support; without backend operator support, operator fusion is meaningless, greatly limiting the scope for fused operators. Even if the operator fusion strategy were user-controllable, inference during backend operator runtime would still be impossible.

[0026] To address the aforementioned issues, this application divides the fusion strategy into a built-in fusion strategy (default fusion strategy) and a user-defined fusion strategy. Users can customize their fusion strategy by modifying the configuration file (which specifies the list of operator types to be fused and the type of operator to be fused). This provides users with a deserialization function for the fusion result, allowing for convenient display and viewing of the fused network graph. This application provides a user-defined backend operator implementation method, allowing users to implement high-performance operators themselves based on their custom fusion strategy. Furthermore, this application provides an automatic code generation mechanism for fusion operators already existing in the inference engine. Through this mechanism, users can rewrite a portion of the network graph's fusion operators via plugins, significantly improving the flexibility of model inference and representing a significant improvement over the plugin approach.

[0027] Figure 2A flowchart of an operator fusion method 200 for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0028] like Figure 2 As shown, in step 201, the deep learning model to be fused with operators is obtained.

[0029] In step 202, a network graph is generated from the deep learning model. The network graph is a directed acyclic graph (DAG) composed of operator connections, which can reflect the various operators of the deep learning model and the connection relationships between operators.

[0030] In step 203, the default fusion strategy can be used to perform operator fusion on the generated network graph. The default fusion strategy (also known as the general fusion strategy) can include commonly used and familiar fusion strategies built into the inference engine that are applicable to all models, such as the fusion of conv operator + batchnorm operator, the fusion of mma operator + batchnorm operator, etc.

[0031] In step 204, the fused network graph is obtained after operator fusion using the default fusion strategy.

[0032] In step 205, a configuration file is obtained. The configuration file includes at least the types of multiple operators to be fused in the user-defined operator fusion strategy and the type of the fused operator.

[0033] In step 206, the user-defined fusion strategy obtained from the configuration file can be used to perform operator fusion on the fused network graph to obtain a re-fused network graph in step 207. The configuration file includes at least the types of the multiple operators to be fused and the types of the fused operators in the user-defined operator fusion strategy. Specifically, in step 206, according to the configuration file, multiple operators with the types of the multiple operators to be fused are fused into an operator with the type of the fused operator.

[0034] Figure 3A A schematic diagram illustrating a first example of fusing multiple operators of the type of multiple operators to be fused into an operator of the type of the fused operator according to an embodiment of the present application, based on a configuration file.

[0035] like Figure 3A As shown, the ONNX model is part of the FaceNet model derived from TensorFlow. Due to compatibility issues during export, the ONNX model contains other operators, resulting in longer inference times, but it is logically equivalent to a Gemm operator.

[0036] Specifically, to fuse multiple operators on the left side of the graph into a single fused operator on the right side, according to embodiments of this application, a configuration file fused_op.json can be written. The configuration file can be a YAML or JSON file. The configuration file can specify the types of operators to be fused, as well as the type of the fused operator. For example, inputting a key-value pair like "Mma": [Matmul,Shape,Cast,Cast,Reshape,BatchNormalization,Reshape] in the configuration file indicates that the types of the multiple operators to be fused in the user-defined operator fusion strategy are Matmul, Shape, Cast, Cast, Reshape, BatchNormalization, and Reshape, and the type of the fused operator is Gemm.

[0037] Therefore, based on the configuration file, specific operators defined by key-value pairs such as "Mma" in the configuration file: [Matmul,Shape,Cast,Cast,Reshape,BatchNormalization,Reshape] can be fused onto the deep learning model.

[0038] In one embodiment, the configuration file may further include the type of deep learning model to be fused in the user-defined operator fusion strategy, wherein fusing the various operators in the network graph of the deep learning model to obtain the fused network graph according to the configuration file includes: fusing the various operators in the network graph of the deep learning model having the type of deep learning model to be fused in the user-defined operator fusion strategy according to the configuration file.

[0039] The configuration file can also include the types of deep learning models to be fused in the user-defined operator fusion strategy. For example, the configuration file can specify the types of models to be fused: ONNX models, TensorFlow *.pb models, or PyTorch *.pt models. For example, with... Figure 3A For example, the configuration file can specify the type of model to be fused as an ONNX model. Thus, based on the configuration file, it can be understood that the deep learning model to be fused in the user-defined operator fusion strategy is an ONNX model. Therefore, based on the configuration file, specific operators defined by key-value pairs such as "Mma" in the configuration file: [Matmul,Shape,Cast,Cast,Reshape,BatchNormalization,Reshape] can be applied to a specific ONNX model for fusion.

[0040] The configuration file can also include the connection relationships between multiple operators to be fused in a user-defined operator fusion strategy, indicating the order of operations between the operators. Figure 3A For example, the connection relationship of multiple operators to be merged in the user-defined operator fusion strategy is the following operation order: from MatMul operator to Shape operator, to Cast operator, to another Cast operator, and then to Reshape operator, and from MatMul operator to Shape operator and then to BatchNormalization operator.

[0041] Figure 3B A schematic diagram illustrating a second example of fusing multiple operators of the types of multiple operators to be fused into an operator of the type of the fused operator according to an embodiment of the present application, based on a configuration file.

[0042] like Figure 3B As shown, the vgg16 model contains the following: Figure 3B The diagram illustrates the operator composition and connection relationships. To maximize computational performance on the graphics processing unit (GPU), the operators need to be fused into... Figure 3B The fused operator is used to obtain optimal performance.

[0043] In this case, you can add the key-value pair "Conv2dBiasReluX2Maxpool" to the configuration file fused_op.json: [Conv, Relu, Conv, Relu, Maxpool]. This means that the types of the multiple operators to be fused in the user-defined operator fusion strategy are Conv, Relu, Conv, Relu, and Maxpool, and the type of the fused operator is Conv2dBiasReluX2Maxpool.

[0044] In one embodiment, the configuration file may further include the type of deep learning model to be fused in a user-defined operator fusion strategy, wherein fusing the various operators in the network graph of the deep learning model to obtain the fused network graph according to the configuration file includes: fusing the various operators in the network graph of the deep learning model having the type of deep learning model to be fused in the user-defined operator fusion strategy according to the configuration file.

[0045] In this way, the configuration file can also specify the type of deep learning model to be fused in the user-defined operator fusion strategy to customize the specific model type to be fused.

[0046] For example, with Figure 3BFor example, the configuration file can also include the type of deep learning model to be fused in the user-defined operator fusion strategy, i.e., the vgg16 model. Thus, based on the configuration file, it can be known that the deep learning model to be fused in the user-defined operator fusion strategy is the vgg16 model. Therefore, based on the configuration file, specific operator fusion defined by key-value pairs such as "Conv2dBiasReluX2Maxpool": [Conv, Relu, Conv, Relu, Maxpool] can be applied to a specific vgg16 model.

[0047] The configuration file can also include the connection relationships between multiple operators to be fused in a user-defined operator fusion strategy, indicating the order of operations between the operators. Figure 3B For example, in a user-defined operator fusion strategy, the connection relationship of multiple operators to be fused is the following operation order: from Conv operator to ReLU operator, then to Conv operator, then to ReLU operator, and then to MaxPool operator.

[0048] Note that the configuration file above, which includes the types of multiple operators to be fused in the user-defined operator fusion strategy, the type of the fused operator, and the type of deep learning model to be fused in the possible user-defined operator fusion strategy, only performs operator fusion based on the network graph. However, whether the actual network graph can achieve true operator fusion computation needs to be specifically implemented through the computation code (or plugin-based computation code) in the inference engine that actually supports this specific operator fusion. The backend operator implementation of the fused network graph according to relevant embodiments of this application will be described later.

[0049] In this way, users can customize their own fusion strategy by modifying the configuration file (which can specify the list of operator types to be fused and the fused operator type), thus achieving flexibility in operator fusion.

[0050] Thus, when the inference engine performs inference, it first performs preprocessing, reading external model files (*.onnx / *.pb / *.pt) and parsing them into a graph structure (Net) of the inference engine's internal network. After that, the inference engine reads the configuration file of the fusion strategy, traverses the network graph, and after finding the operator combination in the network that is in the configuration file, it will fuse the operator combination into a fused operator type. The fused operator type is also determined by the configuration file.

[0051] It is worth noting that the general fusion strategy can also be determined by a configuration file, but this configuration file does not need to be maintained by the user, because it may affect the implementation of the underlying operators.

[0052] Back Figure 2 After obtaining the network graph after fusing the general fusion strategy and the user-defined fusion strategy in step 207, the fused operators may need to be fused again with other operators. Therefore, in optional path 1, the loop can return to the point before the general fusion strategy to perform operator fusion of the general fusion strategy and the user-defined fusion strategy again to obtain a new fused network graph, so as to fuse as many fusionable operators as possible and obtain an optimized network graph to the greatest extent. Alternatively, in optional path 2, the loop can only return to the point before the user-defined fusion strategy to perform operator fusion of the user-defined fusion strategy again. This is because the general fusion strategy usually has a mediocre fusion effect, while the user-defined fusion strategy usually requires fewer loops and less time, but can make the most of the user-defined fusion strategy.

[0053] After a sufficient number of iterations, following step 207 where the fused network graph is obtained using both the general fusion strategy and the user-defined fusion strategy, the fused network graph is serialized to allow the user to visually assess its suitability and identify areas for improvement. This serialization yields a visualized network graph. The visualized network graph is then displayed so the user can further adjust the fusion strategy to meet their expectations.

[0054] Thus, the fusion is carried out recursively in multiple rounds of loops. The operator type after the previous fusion can be used as the operator type to be fused in the next round of loops until there are no more operators to be fused.

[0055] To facilitate manual debugging of the fusion effect, the inference engine can provide a reverse parsing function, allowing users to deserialize the parsed deep learning internal network graph (net) and output it to an external file, which provides visualization capabilities.

[0056] After the operator fusion step of the inference engine, the internal network graph changes according to the fusion rules. This change often differs from the user's expectations. In this case, the user may need to deserialize the current internal network graph to obtain a visualized network graph and the actual fusion effect. Alternatively, the serialized network graph can be saved directly as an intermediate result for subsequent inference or provided to a third party for inference. During re-inference, the steps of specifying the configuration file and operator fusion can be omitted. This deserialization function is performed by the inference engine's parser, which provides forward and reverse parsing functions. Forward parsing parses the external network graph into the internal network graph, while reverse parsing re-serializes the internal network graph back into the external visualized graph.

[0057] Specifically, the fused network graph is serialized, and then in step 208, one or more of the following operations are performed on the serialized network graph: viewing the weight and shape information in the fused network graph, eliminating or modifying one or more weight information in the fused network graph, and eliminating or modifying the shape information of tensors in the fused network graph. In this way, the user can choose to modify the network graph, such as modifying or eliminating weight and tensor information, or simply view it without making any modifications, further customizing the network graph to see the effect of the fused network graph.

[0058] In step 209, an external model file of the serialized network graph, which has undergone one or more operations, is generated to obtain a visualized network graph.

[0059] In step 210, a visualized network diagram is displayed.

[0060] If the user is still not satisfied with the visualized network graph, the method 200 may further include: in step 211, receiving modifications to the network information of the external model file of the network graph; and reloading the fused network graph according to the modifications to obtain the fused network graph in step 207.

[0061] Thus, if the specific fusion strategy does not meet expectations, and the internal fusion strategy may affect the custom fusion strategy, users can monitor the fusion effect. To this end, the inference engine according to this embodiment supports users in serializing the fused graph into an externally visualized model file (such as ONNX). Simultaneously, the serialized model file can be directly used as the fused model for inference without repeated fusion, and the fused model can be used independently of the configuration file. Users can choose whether to extract weight and shape information during serialization, and users can modify some information of the model externally (such as constant values, weight values, etc.), and after modification, directly import it into the inference engine. Using the embodiments of this application, users can monitor, implement, and improve the fusion effect.

[0062] Assuming the user obtains a satisfactory fused network graph, the next step is to consider implementing the network graph desired by the user through backend operators.

[0063] For the fused network graph with the fused operators to function, the actual computation of the fused operators must be implemented. For general fusion strategies, the inference engine pre-implements these fusion operators internally. However, user-defined fusion operators are not implemented internally by the inference engine; users can register their implementations as plugins. The inference engine provides implementation interfaces and registration methods for plugin operators, ensuring that the type of the registered fusion operator matches the type in the configuration file. Users can provide different backend implementations for the fusion operator, such as a Central Processing Unit (CPU) backend or a Graphics Processing Unit (GPU) backend. After the user implements the computation of the specific fusion operator, the inference engine can implement the fusion operator end-to-end.

[0064] The inference engine in this application provides at least two backend support schemes for fusion operators: automatic kernel generation and plugin. If the inference engine already has a corresponding automatic black-box generation mechanism for all basic operator types in the fusion operator list, the inference engine will traverse the fusion operator list to automatically generate the computation code for the entire fusion operator to generate the kernel function. In many cases, the operator list contains one or more operators that are not supported internally by the inference engine. In this case, users can customize the specific implementation of the backend operators. The inference engine provides users with an extensible plugin interface and a plugin operator registration mechanism for this purpose. Users implement several essential functions required by the inference engine (such as execute and shapeInference functions) according to the custom fusion operator type, and register the implemented plugin operators with the inference engine to complete the registration. If users are not satisfied with the performance of the automatically generated kernel function, they can suppress the inference method of the internal automatic kernel function generator by using a flag, and use the plugin method for inference.

[0065] Specifically, such as Figure 2As shown, method 200 may further include: in step 212, implementing backend operators on the fused network graph. This backend operator implementation can include several methods: for example, automatic generation by an automatic kernel function generator, implementation by writing a user-defined fusion operator implementation plugin, or implementation using existing internal operator implementations. Specifically, if automatic generation by an automatic kernel function generator is used, all operators before fusion have registered with the automatic kernel function generator, thus automatically generating a fused kernel function. These methods can be used individually or in combination. For example, all operators can be automatically generated by an automatic kernel function generator, or all operators can be implemented by writing a user-defined fusion operator implementation plugin, or all operators can be implemented using existing internal operator implementations, or operators that can be automatically generated by an automatic kernel function generator can be automatically generated, while other operators can be implemented by writing a user-defined fusion operator implementation plugin, and so on. The choice can be made based on the user's requirements for operator implementation and desired results.

[0066] Specifically, the steps for implementing backend operators on the fused network graph may include one or more of the following:

[0067] like Figure 2 As shown, in step 213, it is checked whether all operators in the fused network graph can have their computational code generated by an automatic kernel function generator (e.g., Auto Kernel) (checking if the automatic kernel function generator contains single operators from the fused network graph). The automatic kernel function generator is an automatic operator optimization tool that can automatically optimize scheduling strategies and automatically generate low-level optimization code. If yes, then in step 214, the computational code of the operators is automatically generated by the automatic kernel function generator; if not, then in step 215, the computational code of operators that cannot have their computational code generated by the automatic kernel function generator is obtained by writing a user-defined fusion operator implementation plugin.

[0068] User-defined fusion operator implementation plugins can be written in C++, and can be compiled from source code separately, linked into a dynamic library, and then loaded and imported when the operator backend implementation is needed.

[0069] Alternatively, if the operators in the merged network graph can be implemented by writing a user-defined fusion operator implementation plugin (regardless of whether the operator's computational code can be automatically generated by an automatic kernel function generator or obtained using existing internal operator implementations), then the computational code for each operator in the merged network graph will be obtained by writing a user-defined fusion operator implementation plugin. In this way, the user-defined fusion operator implementation plugin will be better suited to the performance of the operator implementation and may perform better than the computational code generated by the automatic kernel function generator; therefore, the user-defined fusion operator implementation plugin should be preferred.

[0070] Alternatively, in step 216, existing internal operator implementations can be used to obtain the computational code corresponding to the operators in the fused network graph. There may already be some internal operator implementations with computational code within the inference engine; these can be used directly to complete the operator backend implementation.

[0071] like Figure 3A For example, adding a backend operator implementation to the Mma operator, implementing the necessary functions such as shapeInference and execute, and registering them in the inference engine; running the inference engine according to this embodiment, these operators on the left will be merged into a single Mma operator. According to this embodiment, the actual fusion effect can be viewed by calling the serialization function of the internal network (e.g., ...). Figure 3A (On the right), it is exactly as the user expected. In this way, the program can run end-to-end, maximizing inference performance.

[0072] like Figure 3B For example, a high-performance implementation of the Conv2dBiasReluX2Maxpool operator is added, along with necessary implementations such as shape calculation. The fused ONNX graph is viewed using the inference engine's internal network visualization tool, as shown on the right. Figure 3B As shown on the right, it matches the user's expectations. If it does not match, you can continue to adjust the configuration file. In this way, you can obtain the newly fused VGG16 network for end-to-end inference and check the performance improvement.

[0073] Thus, this solution provides an inference engine implementation scheme with user-configurable fusion strategies. Users can adopt specific custom fusion strategies based on the characteristics of each model to achieve maximum performance optimization. It provides an internal network serialization method for the inference engine to reverse-parse the internal network into a visual external file. Users can use the aforementioned tools to obtain the effect diagram of each fusion step, helping them debug the fusion strategy. It provides a backend interface for users to implement specific fusion operators, enabling operators to not only fuse but also run. This maximizes the fusion degree of specific models, simplifies the models, and reduces the number of model operators. Maximum fusion can also be achieved for user-defined operators (customops). It provides users with an offline visualization interface for fusion effects, facilitating viewing and debugging of different fusion strategies. The fused offline graph is saved as an intermediate result independently of the fusion strategy configuration file, eliminating the need for re-fusion during re-parse. It provides users with a convenient backend implementation and registration mechanism for fusion operators. User-defined backend implementations of fusion operators bypass a large amount of redundant computation and directly obtain the results. Users can write their own high-performance versions for specific operators, facilitating the expansion of user functionality. For operators not supported by the inference framework, model adaptation can be completed quickly, greatly improving the inference engine's support for model generalization.

[0074] Figure 4 A flowchart of an operator fusion method 400 for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0075] like Figure 4 As shown, the operator fusion method 400 of the inference engine of the deep learning model includes: step 401, obtaining a configuration file, which includes at least the types of multiple operators to be fused in the user-defined operator fusion strategy and the type of the fused operator; step 402, according to the configuration file, fusing each operator in the network graph of the deep learning model to obtain the fused network graph.

[0076] In this way, users can customize their own fusion strategy by modifying the configuration file (which can specify the list of operator types to be fused and the fused operator type), thus achieving flexibility in operator fusion.

[0077] In one embodiment, fusing the various operators in the network graph of the deep learning model to obtain a fused network graph according to the configuration file includes: fusing multiple operators with multiple operator types to be fused into an operator with the fused operator type.

[0078] In this way, the configuration file can also specify the type of deep learning model to be fused in the user-defined operator fusion strategy to customize the specific model type to be fused.

[0079] In one embodiment, the configuration file may also include the connection relationship of multiple operators to be fused in a user-defined operator fusion strategy, indicating the order of operations between the operators.

[0080] In one embodiment, the configuration file further includes the type of deep learning model to be fused in a user-defined operator fusion strategy, wherein fusing the various operators in the network graph of the deep learning model to obtain the fused network graph according to the configuration file includes: fusing the various operators in the network graph of the deep learning model having the type of deep learning model to be fused in the user-defined operator fusion strategy according to the configuration file.

[0081] In one embodiment, method 400 further includes fusing the various operators in the network graph of the deep learning model according to the general operator fusion strategy built into the inference engine.

[0082] In one embodiment, method 400 further includes: fusing the fused network graph once or multiple times using a user-defined operator fusion strategy and / or a general operator fusion strategy to obtain a new fused network graph.

[0083] Thus, the fusion is carried out recursively in multiple rounds of loops. The operator type after the previous fusion can be used as the operator type to be fused in the next round of loops until there are no more operators to be fused.

[0084] In one embodiment, method 400 further includes: serializing the fused network graph to obtain a visualized network graph; and displaying the visualized network graph.

[0085] In one embodiment, serializing the fused network graph to obtain a visualized network graph includes: serializing the fused network graph; receiving one or more of the following operations performed on the serialized network graph: viewing weight information and shape information in the fused network graph, eliminating or modifying one or more weight information in the fused network graph, eliminating or modifying the shape information of tensors in the fused network graph; generating an external model file of the serialized network graph that has undergone one or more operations to obtain a visualized network graph; and displaying the visualized network graph.

[0086] Therefore, if the specific fusion strategy does not meet expectations, and the internal fusion strategy may affect the custom fusion strategy, users can monitor the fusion effect.

[0087] In one embodiment, method 400 further includes: receiving modifications to network information in an external model file of the network graph; and reloading the fused network graph based on the modifications.

[0088] Thus, by utilizing the embodiments of this application, users can monitor, implement, and improve the fusion effect.

[0089] In one embodiment, method 400 further includes: implementing backend operators on the fused network graph.

[0090] In one embodiment, implementing backend operators on the fused network graph includes one or more of the following: checking whether all operators in the fused network graph can have their computational code generated by an automatic kernel function generator; if so, automatically generating the computational code for the operators using the automatic kernel function generator; if not, obtaining the computational code for operators that cannot have their computational code generated by the automatic kernel function generator by writing a user-defined fusion operator implementation plugin; if the operators in the fused network graph can be implemented by writing a user-defined fusion operator implementation plugin, obtaining the computational code for each operator in the fused network graph by writing a user-defined fusion operator implementation plugin; and using existing internal operator implementations to obtain the computational code corresponding to the operators in the fused network graph.

[0091] In this way, user-defined fusion operators and other backend implementations can be achieved.

[0092] Thus, this solution provides an inference engine implementation scheme with user-configurable fusion strategies. Users can adopt specific custom fusion strategies based on the characteristics of each model to achieve maximum performance optimization. It provides an internal network serialization method for the inference engine to reverse-parse the internal network into a visual external file. Users can use the aforementioned tools to obtain the effect diagram of each fusion step, helping them debug the fusion strategy. It provides a backend interface for users to implement specific fusion operators, enabling operators to not only fuse but also run. This maximizes the fusion degree of specific models, simplifies the models, and reduces the number of model operators. Maximum fusion can also be achieved for user-defined operators. It provides users with an offline visualization interface for fusion effects, facilitating viewing and debugging of different fusion strategies. The fused offline graph is saved as an intermediate result independently of the fusion strategy configuration file, eliminating the need for re-fusion during re-parse. It provides users with a convenient backend implementation and registration mechanism for fusion operators. User-defined backend implementations of fusion operators bypass a large amount of redundant computation and directly obtain the results. Users can write their own high-performance versions for specific operators, facilitating the expansion of user functionality. For operators not supported by the inference framework, model adaptation can be completed quickly, greatly improving the inference engine's support for model generalization.

[0093] Figure 5 A block diagram of an operator fusion system 500 for an inference engine of a deep learning model according to an embodiment of this application is shown.

[0094] like Figure 5 As shown, the operator fusion system 500 of the inference engine of the deep learning model includes: a configuration device 501, configured to obtain a configuration file, the configuration file including at least the types of multiple operators to be fused in the user-defined operator fusion strategy and the type of the fused operator; and a fusion device 502, configured to fuse the various operators in the network graph of the deep learning model according to the configuration file to obtain the fused network graph.

[0095] In this way, the configuration file can also specify the type of deep learning model to be fused in the user-defined operator fusion strategy to customize the specific model type to be fused.

[0096] In one embodiment, the fusion device 502 is configured to fuse multiple operators of the type of multiple operators to be fused into an operator of the type of the fused operator.

[0097] In one embodiment, the configuration file further includes the type of deep learning model to be fused in a user-defined operator fusion strategy, wherein the fusion device 502 is configured to fuse the various operators in the network graph of the deep learning model having the type of deep learning model to be fused in the user-defined operator fusion strategy, according to the configuration file.

[0098] In one embodiment, the configuration file also includes the connection relationship of multiple operators to be fused in a user-defined operator fusion strategy, indicating the order of operations between the operators.

[0099] In one embodiment, the operator fusion system 500 further includes a general fusion device configured to fuse the various operators in the network graph of the deep learning model according to a general operator fusion strategy built into the inference engine.

[0100] In one embodiment, the operator fusion system 500 further includes a cyclic device configured to perform user-defined operator fusion strategy and / or general operator fusion strategy on the fused network graph one or more times to obtain a new fused network graph.

[0101] Thus, the fusion is carried out recursively in multiple rounds of loops. The operator type after the previous fusion can be used as the operator type to be fused in the next round of loops until there are no more operators to be fused.

[0102] In one embodiment, the operator fusion system 500 further includes: a serialization device configured to serialize the fused network graph to obtain a visualized network graph; and a display device configured to display the visualized network graph.

[0103] In one embodiment, the serialization apparatus is configured to: serialize the fused network graph; receive one or more of the following operations performed on the serialized network graph: viewing weight information and shape information in the fused network graph, eliminating or modifying one or more weight information in the fused network graph, eliminating or modifying the shape information of tensors in the fused network graph; generating an external model file of the serialized network graph that has undergone one or more operations to obtain a visualized network graph; and displaying the visualized network graph.

[0104] Therefore, if the specific fusion strategy does not meet expectations, and the internal fusion strategy may affect the custom fusion strategy, users can monitor the fusion effect.

[0105] In one embodiment, the operator fusion system 500 further includes: a modification device configured to receive modifications to network information of an external model file of the network graph; and a reloading device configured to reload the fused network graph according to the modifications.

[0106] Thus, by utilizing the embodiments of this application, users can monitor, implement, and improve the fusion effect.

[0107] In one embodiment, the operator fusion system 500 further includes: an implementation device configured to perform back-end operator implementation on the fused network graph.

[0108] In one embodiment, the implementation apparatus is configured to: check whether all operators in the fused network graph can have computational code generated by an automatic kernel function generator; if so, automatically generate computational code for the operators using the automatic kernel function generator; if not, obtain computational code for operators that cannot have computational code generated by the automatic kernel function generator by writing a user-defined fusion operator implementation plugin; if operators in the fused network graph can be implemented by writing a user-defined fusion operator implementation plugin, obtain computational code for each operator in the fused network graph by writing a user-defined fusion operator implementation plugin; and obtain computational code corresponding to the operators in the fused network graph using existing internal operator implementations.

[0109] In this way, user-defined fusion operators and other backend implementations can be achieved.

[0110] Thus, this solution provides an inference engine implementation scheme with user-configurable fusion strategies. Users can adopt specific custom fusion strategies based on the characteristics of each model to achieve maximum performance optimization. It provides an internal network serialization method for the inference engine to reverse-parse the internal network into a visual external file. Users can use the aforementioned tools to obtain the effect diagram of each fusion step, helping them debug the fusion strategy. It provides a backend interface for users to implement specific fusion operators, enabling operators to not only fuse but also run. This maximizes the fusion degree of specific models, simplifies the models, and reduces the number of model operators. Maximum fusion can also be achieved for user-defined operators. It provides users with an offline visualization interface for fusion effects, facilitating viewing and debugging of different fusion strategies. The fused offline graph is saved as an intermediate result independently of the fusion strategy configuration file, eliminating the need for re-fusion during re-parse. It provides users with a convenient backend implementation and registration mechanism for fusion operators. User-defined backend implementations of fusion operators bypass a large amount of redundant computation and directly obtain the results. Users can write their own high-performance versions for specific operators, facilitating the expansion of user functionality. For operators not supported by the inference framework, model adaptation can be completed quickly, greatly improving the inference engine's support for model generalization.

[0111] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of this application is shown.

[0112] The electronic device may include a processor (H1); and a storage medium (H2) coupled to the processor (H1) and storing computer-executable instructions therein for performing the steps of various methods of the embodiments of this application when executed by the processor.

[0113] The processor (H1) may include, but is not limited to, one or more processors or microprocessors.

[0114] Storage media (H2) may include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CD-ROMs, DVD-ROMs, Blu-ray discs, etc.).

[0115] In addition, the electronic device may also include a data bus (H3), an input / output (I / O) bus (H4), a display (H5), and input / output devices (H6) (e.g., keyboard, mouse, speaker, etc.).

[0116] The processor (H1) can communicate with external devices (H5, H6, etc.) via the I / O bus (H4) through a wired or wireless network (not shown).

[0117] The storage medium (H2) may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described in this technology when executed by the processor (H1).

[0118] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0119] Figure 7 A schematic diagram of a non-transitory computer-readable storage medium according to an embodiment of this application is shown.

[0120] like Figure 7 As shown, the computer-readable storage medium 720 stores instructions, such as computer-readable instruction 710. When the computer-readable instruction 710 is executed by a processor, the various methods described above can be performed. The computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the computer-readable storage medium 720 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instruction 710 stored on the computer-readable storage medium 720, the various methods described above can be performed.

[0121] Thus, this solution provides an inference engine implementation scheme with user-configurable fusion strategies. Users can adopt specific custom fusion strategies based on the characteristics of each model to achieve maximum performance optimization. It provides an internal network serialization method for the inference engine to reverse-parse the internal network into a visual external file. Users can use the aforementioned tools to obtain the effect diagram of each fusion step, helping them debug the fusion strategy. It provides a backend interface for users to implement specific fusion operators, enabling operators to not only fuse but also run. This maximizes the fusion degree of specific models, simplifies the models, and reduces the number of model operators. Maximum fusion can also be achieved for user-defined operators. It provides users with an offline visualization interface for fusion effects, facilitating viewing and debugging of different fusion strategies. The fused offline graph is saved as an intermediate result independently of the fusion strategy configuration file, eliminating the need for re-fusion during re-parse. It provides users with a convenient backend implementation and registration mechanism for fusion operators. User-defined backend implementations of fusion operators bypass a large amount of redundant computation and directly obtain the results. Users can write their own high-performance versions for specific operators, facilitating the expansion of user functionality. For operators not supported by the inference framework, model adaptation can be completed quickly, greatly improving the inference engine's support for model generalization.

[0122] Of course, the specific embodiments described above are merely examples and not limitations. Those skilled in the art can combine and integrate some steps and devices from the various embodiments described separately above to achieve the effects of this application based on the concept of this application. Such combined and integrated embodiments are also included in this application, but will not be described one by one here.

[0123] Note that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations; these details do not restrict this application from being implemented using the aforementioned specific details.

[0124] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0125] The flowcharts and method descriptions in this disclosure are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the given order. As those skilled in the art will recognize, the steps in the above embodiments can be performed in any order. Words such as "then," "next," etc., are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to a singular element, such as the use of the articles "a," "one," or "the," is not to be construed as limiting that element to the singular.

[0126] Furthermore, the steps and apparatus in the various embodiments herein are not limited to any one embodiment. In fact, new embodiments can be conceived by combining relevant steps and apparatus in the various embodiments herein based on the concepts of this application, and these new embodiments are also included within the scope of this application.

[0127] The various operations of the methods described above can be performed by any suitable means capable of performing the corresponding functions. Such means may include various hardware and / or software components and / or modules, including but not limited to hardware circuits, application-specific integrated circuits (ASICs), or processors.

[0128] The various exemplified logic blocks, modules, and circuits described herein can be implemented or performed using a general-purpose processor, digital signal processor (DSP), ASIC, field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor, but alternatively, it can be any commercially available processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, a microprocessor cooperating with a DSP core, or any other such configuration.

[0129] The steps of the methods or algorithms described in this disclosure can be directly embedded in hardware, in a software module executed by a processor, or a combination of both. The software module can reside in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. The storage medium can be coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integral with the processor. The software module can be a single instruction or many instructions, and can be distributed across several different code segments, different programs, and across multiple storage media.

[0130] The methods disclosed herein include actions for implementing the described methods. The methods and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims.

[0131] The above functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored as instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital universal disc (DVD), floppy disk, and Blu-ray disc, wherein a disc typically magnetically reproduces data, while a disc optically reproduces data using lasers.

[0132] Therefore, a computer program product can perform the operations given herein. For example, such a computer program product can be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which can be executed by a processor to perform the operations described herein. The computer program product may include packaging materials.

[0133] Software or instructions can also be transmitted via a transmission medium. For example, software can be transmitted from a website, server, or other remote source using transmission media such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave.

[0134] Furthermore, modules and / or other suitable means for carrying out the methods and techniques described herein can be downloaded and / or obtained by user terminals and / or base stations as appropriate. For example, such a device can be coupled to a server to facilitate the transmission of means for carrying out the methods described herein. Alternatively, the various methods described herein can be provided via storage components (e.g., RAM, ROM, physical storage media such as CDs or floppy disks) so that user terminals and / or base stations can obtain the various methods when coupled to the device or when storage components are provided to the device. Furthermore, any other suitable techniques for providing the methods and techniques described herein to the device can be utilized.

[0135] Other examples and implementations are within the scope and spirit of this disclosure and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Features implementing the functions can also be physically located in various places, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the described examples are preferred or better than other examples.

[0136] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0137] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0138] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An operator fusion method for the inference engine of a deep learning model, comprising: Obtain the configuration file, which includes at least the types of multiple operators to be fused in the user-defined operator fusion strategy and the type of the fused operator; According to the configuration file, the various operators in the network graph of the deep learning model are fused to obtain the fused network graph; The fused network graph is then serialized. The system receives one or more of the following operations on the serialized network graph: viewing the weight information and shape information in the fused network graph, eliminating or modifying one or more weight information in the fused network graph, and eliminating or modifying the shape information of tensors in the fused network graph. Generate an external model file of the serialized network graph that has undergone one or more operations to obtain a visualized network graph; Display the visualized network graph; Receive modifications to the network information of the external model file of the network graph; The modified fused network graph is obtained by reloading the fused network graph according to the modification. The deep learning model, implemented by operators in the fused network graph, is loaded for inference and prediction. The backend operator is implemented for the fused network graph; The implementation of backend operators on the fused network graph includes one or more of the following: Check whether all operators in the fused network graph can have their computation code generated by the automatic kernel function generator. If so, the computation code of the operator is automatically generated by the automatic kernel function generator. If not, the computation code of the operator that cannot have its computation code generated by the automatic kernel function generator is obtained by writing a user-defined fusion operator implementation plugin. If the operators in the fused network graph can be implemented by writing a user-defined fusion operator implementation plugin, then the calculation code of each operator in the fused network graph can be obtained by writing a user-defined fusion operator implementation plugin. The computation code corresponding to the operator in the fused network graph is obtained by utilizing the existing internal operator implementation.

2. The method according to claim 1, wherein, The step of fusing the various operators in the network graph of the deep learning model according to the configuration file to obtain the fused network graph includes: Multiple operators of the types to be merged are merged into an operator of the type of the merged operator.

3. The method according to claim 1, wherein, The configuration file also includes the type of deep learning model to be fused in the user-defined operator fusion strategy. The process of fusing the various operators in the network graph of the deep learning model to obtain the fused network graph according to the configuration file includes: fusing the various operators in the network graph of the deep learning model that has the type of deep learning model to be fused in the user-defined operator fusion strategy according to the configuration file.

4. The method according to claim 1, wherein, The configuration file also includes the connection relationships of multiple operators to be merged in the user-defined operator fusion strategy.

5. The method according to claim 1, further comprising: Based on the general operator fusion strategy built into the inference engine, the various operators in the network graph of the deep learning model are fused.

6. The method according to any one of claims 1-5, further comprising: The fused network graph is then fused once or multiple times using a user-defined operator fusion strategy and / or a general operator fusion strategy to obtain a new fused network graph.

7. An operator fusion system for an inference engine of a deep learning model, comprising: The configuration device is configured to obtain a configuration file, which includes at least the types of multiple operators to be fused in a user-defined operator fusion strategy and the type of the fused operator; The fusion device is configured as follows: According to the configuration file, the various operators in the network graph of the deep learning model are fused to obtain the fused network graph; The fused network graph is then serialized. The system receives one or more of the following operations on the serialized network graph: viewing the weight information and shape information in the fused network graph, eliminating or modifying one or more weight information in the fused network graph, and eliminating or modifying the shape information of tensors in the fused network graph. Generate an external model file of the serialized network graph that has undergone one or more operations to obtain a visualized network graph; Display the visualized network graph; Receive modifications to the network information of the external model file of the network graph; The modified fused network graph is obtained by reloading the fused network graph according to the modification. The deep learning model, implemented by operators in the fused network graph, is loaded for inference and prediction. The backend operator is implemented for the fused network graph; The implementation of backend operators on the fused network graph includes one or more of the following: Check whether all operators in the fused network graph can have their computation code generated by the automatic kernel function generator. If so, the computation code of the operator is automatically generated by the automatic kernel function generator. If not, the computation code of the operator that cannot have its computation code generated by the automatic kernel function generator is obtained by writing a user-defined fusion operator implementation plugin. If the operators in the fused network graph can be implemented by writing a user-defined fusion operator implementation plugin, then the calculation code of each operator in the fused network graph can be obtained by writing a user-defined fusion operator implementation plugin. The computation code corresponding to the operator in the fused network graph is obtained by utilizing the existing internal operator implementation.

8. An electronic device, comprising: Memory, used to store instructions; A processor for reading instructions from the memory and executing the method as described in any one of claims 1-6.

9. A non-temporary storage medium having instructions stored thereon, in, When the instruction is read by the processor, it causes the processor to execute the method as described in any one of claims 1-6.

Images