A middleware execution method, system, device, and medium for a processor platform

Abstract

This invention discloses a middleware runtime method, system, device, and medium for a processor platform, belonging to the field of electronic information technology applications. The method includes: acquiring input data; performing memory serialization of an ONNX model based on the input data; decomposing the memory-serialized ONNX model into multiple subgraphs using a model segmenter; allocating the multiple subgraphs to different model executors through a Provider registrar; the model executors running the model, performing inference calculations, and generating processed output data; and outputting the processed output data from the system. The runtime middleware significantly improves the computational efficiency and resource utilization of intelligent processors by optimizing data flow and control flow. This invention aims to achieve efficient and flexible program execution and resource management to adapt to increasingly complex artificial intelligence application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology applications, and specifically relates to a middleware operation method, system, device and medium for a processor platform. Background Technology

[0002] In the field of electronic information technology, with the increasing demand for artificial intelligence and high-performance computing, intelligent processor platforms such as the Cambricon series processors are being used more and more widely in fields such as deep learning and machine learning. However, existing technologies have some limitations in achieving cross-platform, efficient program execution and resource management. In particular, on the Cambricon Processor platform, although its hardware performance is powerful, it lacks a runtime middleware that can fully realize its potential to support mainstream artificial intelligence programming frameworks, such as ONNXRuntime.

[0003] While the existing ONNXRuntime inference engine supports various deep learning frameworks and hardware platforms, its direct support for the Cambricon intelligent processor platform is limited. This leads to numerous challenges for developers when deploying and running AI applications, including but not limited to the complexity of model deployment, inefficient use of hardware resources, and cross-platform compatibility issues. Furthermore, existing solutions typically require developers to delve into the low-level details of the processor, which not only increases development difficulty but also limits the flexibility of algorithm and application development. Summary of the Invention

[0004] The purpose of this invention is to provide a middleware operation method, system, device, and readable storage medium for a processor platform, solving the problems of high development difficulty and limited flexibility in algorithm and application development in the prior art; this invention significantly improves the computational efficiency and resource utilization of the Cambricon intelligent processor by optimizing the memory serialization, model segmentation and allocation of the ONNX model, as well as efficient inference computation.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A middleware execution method for a processor platform, comprising:

[0007] Get the input data;

[0008] Memory serialization of the ONNX model based on the input data;

[0009] The ONNX model, after being serialized in memory, is decomposed into multiple subgraphs using a model segmenter.

[0010] Multiple subgraphs are assigned to different model executors via a Provider registrar;

[0011] The model executor runs the model, performs inference calculations, and generates processed output data.

[0012] The processed output data is output from the system.

[0013] Preferably, before performing memory serialization of the ONNX model based on the input data, the method further includes:

[0014] Determine whether the current model file in the input data has been compiled. If not, compile the ONNX model file.

[0015] Furthermore, the compilation of the ONNX model file includes:

[0016] Construct a Graph class, and use the Status ParserONNXModel function to convert the serialized_string format *.onnx model into a Network structure, and then construct a me::Graph object from the ONNX model.

[0017] Furthermore, if the current model file in the input data has already been compiled, the compiled file "*.mm" is read into memory through the ModelLoadFromFile function, a me::ModelDesc object is constructed, the number of input and output data is set, and the compiled model in memory is bound to the model ID through me::ModelDescContainer, and then memory serialization is performed directly.

[0018] Preferably, the in-memory serialization of the ONNX model based on the input data includes:

[0019] Configure the builder, using the me::ModelBufferData object to store the serialized model data and size in memory, named "*.mm"; manage it through the me::ModelDescContainer object, bind it to the model ID, and convert the model into a format suitable for inference.

[0020] Preferably, after acquiring the input data, the input data is preprocessed to meet the input requirements of the model.

[0021] Preferably, the model executor runs the model and performs inference calculations, including:

[0022] To perform model inference, the cnrtQueueCreate function creates a Cambricon intelligent processor runtime queue_ to manage the queue for model execution.

[0023] Call the Enqueue method of the context object in CambriconModelPreparation, passing the input prepare.inputSet_, the output prepare.outputSet_, and the queue_ to the Enqueue method to schedule the actual execution of the model.

[0024] A middleware runtime system for a processor platform, comprising:

[0025] The data acquisition module is used to acquire input data;

[0026] The ONNX model module is used for in-memory serialization of ONNX models based on input data;

[0027] The model segmentation module is used to decompose the memory-serialized ONNX model into multiple subgraphs through the model segmenter, and to assign the multiple subgraphs to different model executors through the provider registrar.

[0028] The model execution module is used to run the model, perform inference calculations, generate processed output data, and output the processed output data from the system.

[0029] A computer device, comprising:

[0030] A processor is used to execute computer programs;

[0031] A computer-readable storage medium storing a computer program, which, when executed by the processor, performs a middleware execution method for a processor platform as described in any of the preceding claims.

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a middleware operation method for a processor platform as described in any of the preceding claims.

[0033] Compared with the prior art, the present invention has the following beneficial technical effects:

[0034] This invention provides a middleware runtime method for a processor platform. The runtime middleware significantly improves the computational efficiency and resource utilization of intelligent processors by optimizing data and control flow. It also provides a complete set of API interfaces, enabling developers to easily deploy and run artificial intelligence applications on the Cambricon intelligent processor platform. This runtime middleware provides a powerful runtime environment for the intelligent processor platform, allowing developers to focus more on algorithm and application development without delving into the processor's low-level details, thereby accelerating the research and application of artificial intelligence technology. This invention aims to achieve efficient and flexible program execution and resource management to adapt to increasingly complex artificial intelligence application scenarios. The runtime middleware proposed in this invention is particularly suitable for Cambricon intelligent processors, fully leveraging their advantages in deep learning, machine learning, and other high-performance computing tasks, while also supporting mainstream AI programming frameworks such as ONNXRuntime. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the ONNXRuntime architecture for implementing deep learning models in Embodiment 1 of the present invention.

[0036] Figure 2 This is a flowchart illustrating the operation of the middleware software in Embodiment 2 of the present invention;

[0037] Figure 3 This is a flowchart illustrating the deployment, compilation, and execution of the deep learning model on the Cambricon intelligent processor platform in Embodiment 3 of the present invention. Detailed Implementation

[0038] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0039] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0040] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0041] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a communication connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0042] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0043] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0044] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0045] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0046] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0047] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] Example 1

[0049] This invention provides a middleware execution method for a processor platform, comprising:

[0050] The system acquires input data; performs memory serialization of the ONNX model based on the input data; decomposes the memory-serialized ONNX model into multiple subgraphs using a model segmenter; assigns the multiple subgraphs to different model executors using a Provider registrar; the model executors run the model, perform inference calculations, and generate processed output data; and outputs the processed output data from the system.

[0051] The runtime middleware of this invention significantly improves the computational efficiency and resource utilization of intelligent processors by optimizing data and control flows. It also provides a complete set of API interfaces, enabling developers to easily deploy and run artificial intelligence applications on the Cambricon intelligent processor platform.

[0052] This invention, using the Cambricon intelligent processor as an example, presents a runtime middleware designed for the Cambricon intelligent processor platform. It aims to achieve efficient and flexible program execution and resource management to adapt to increasingly complex artificial intelligence application scenarios. The runtime middleware proposed in this invention is particularly suitable for the Cambricon intelligent processor, fully leveraging its advantages in deep learning, machine learning, and other high-performance computing tasks, while also supporting mainstream AI programming frameworks such as ONNXRuntime.

[0053] This invention implements the runtime middleware for the Cambricon intelligent processor platform based on ONNXRuntime, an open-source deep learning inference engine used to efficiently execute deep learning models on different hardware and platforms.

[0054] Architecture diagram as follows Figure 1This paper demonstrates a cross-platform ONNX model execution environment. Input data is first fed into the system, followed by in-memory serialization of the ONNX model, ready for execution. A model splitter decomposes the serialized ONNX model into multiple subgraphs, which are then assigned to different model executors via a provider registrar. The model executors are responsible for interacting with specific execution providers (such as CPUs or Cambricon) to execute the various parts of the model on the appropriate hardware. Execution providers are components that interact directly with the hardware platform, leveraging hardware features to optimize and execute the model. Finally, the processed output data is output from the system. This entire process ensures that ONNX models can run efficiently and compatiblely on CPUs and specific hardware platforms.

[0055] In ONNXRuntime, "EP" stands for Execution Provider, a component used to execute deep learning model inference on different hardware platforms. Execution Providers enable ONNXRuntime to run models on various hardware and accelerators. However, each EP is developed for a specific hardware or accelerator, and cannot be directly used with the model processing solutions provided by Cambricon's intelligent processors. Middleware software is needed to handle model calls and input / output interactions, thereby enabling model deployment on the Cambricon intelligent processor platform.

[0056] This invention supports the Cambricon intelligent processor hardware platform by adding a new EP (Execution Module) to ONNXRuntime. The EP is an abstraction layer that provides multiple interfaces for the execution model.

[0057] (1) Create a new EP class: This step is to define a new EP class (CAMBRICONExecutionProvider), which inherits the IExecutionProvider interface provided by ONNXRuntime and implements its virtual functions so that it can be registered and called at runtime.

[0058] (2) Create a factory class that inherits from the IExecutionProviderFactory interface and implements the CreateProvider() function so that EP objects can be created in the session options.

[0059] (3) Implement the IExecutionProviderfactory interface: used to create and register instances of EP. It has a CreateProvider method that returns a pointer to the IExecutionProvider interface.

[0060] (4) Implement the IExecutionProvider interface: It is used to define the main logic of EP, including initialization, memory allocation to create computation flow, execution of computation graph, etc. Among them, the CAMBRICONExecutionProvider::Compile function is responsible for completing the compilation and execution process of the model. The entire middleware is implemented around this interface.

[0061] like Figure 2 As shown, its content can be summarized into the following core steps:

[0062] 1. Determine if the current model file has been compiled. If it has, use the ModelLoadFromFile function to read the compiled file "*.mm" into memory, construct a me::ModelDesc object, set the number of input and output data, and finally use me::ModelDescContainer to bind the compiled model in memory with the model ID for easy retrieval later. Proceed directly to step 3.

[0063] 2. If the current model file has not been compiled, the ONNX model file needs to be compiled. First, a Graph class is constructed to describe the graph structure of the Cambricon Intelligent Processor MagicMind engine. The Network class is constructed by using the StatusParserONNXModel(std::string string_model, me::Graph&graph) function to construct the *.onnx model in serialized_string format into a Network structure (Graph) supported by the Cambricon Intelligent Processor, and then constructs a me::Graph object from the ONNX model.

[0064] 2. Model Memory Serialization

[0065] Secondly, configure the builder, using a `me::ModelBufferData` object to store the serialized model data and size in memory, named "*.mm". Finally, manage it through a `me::ModelDescContainer` object, binding it to the model ID for easy retrieval later.

[0066] 3. Input / output mapping construction:

[0067] Construct an ONNXRuntime::cambricon::CambriconModelPreparation object, build the input map and output map, mapping the node's input and output names to their indices. Prepare the input data, including copying it from host memory to device memory.

[0068] 4. Perform model inference. Use the cnrtQueueCreate function to create a Cambricon Runtime queue_ to manage the queue for model execution.

[0069] Call the Enqueue method of the context object in CambriconModelPreparation, passing the input prepare.inputSet_, the output prepare.outputSet_, and the queue_ to the Enqueue method to schedule the actual execution of the model.

[0070] 5. Process output data, including copying it from device memory back to host memory.

[0071] This invention also provides a middleware runtime system for a processor platform, comprising:

[0072] The data acquisition module is used to acquire input data;

[0073] The ONNX model module is used for in-memory serialization of ONNX models based on input data;

[0074] The model segmentation module is used to decompose the memory-serialized ONNX model into multiple subgraphs through the model segmenter, and to assign the multiple subgraphs to different model executors through the provider registrar.

[0075] The model execution module is used to run the model, perform inference calculations, generate processed output data, and output the processed output data from the system.

[0076] The purpose of this invention is to overcome the limitations of existing technologies on the Cambricon intelligent processor platform, particularly in terms of runtime middleware. We aim to develop an innovative runtime middleware that not only significantly improves the computational efficiency and resource utilization of the Cambricon intelligent processor when processing deep learning models, but also seamlessly supports mainstream artificial intelligence programming frameworks such as ONNXRuntime. This middleware will provide developers with a comprehensive and powerful tool, enabling them to more easily deploy and execute complex artificial intelligence applications on the Cambricon intelligent processor platform.

[0077] Through this invention, we aim to simplify the deployment and operation of models, lower the technical threshold, and allow developers to focus more on algorithm innovation and application development rather than getting bogged down in hardware complexities. Furthermore, this middleware will enhance the cross-platform compatibility of models on Cambricon's intelligent processors, ensuring they can run efficiently in different hardware and software environments while maintaining high performance.

[0078] The runtime middleware technology of this invention provides a powerful operating environment for the Cambricon intelligent processor platform, enabling developers to focus more on the development of algorithms and applications without having to delve into the underlying details of the processor, thereby accelerating the research and application of artificial intelligence technology.

[0079] Example 2

[0080] like Figure 3 As shown, the deployment, compilation, and execution process of deep learning models on the Cambricon processor platform is described in detail below:

[0081] (1) Constructing the input queue and its shape: Prepare the input data, construct the data queue, and determine the shape of the data.

[0082] (2) Input data preprocessing: Perform necessary preprocessing on the input data to meet the input requirements of the model.

[0083] (3) Construct the address corresponding to the output data: Allocate memory address for the output data so that the results can be stored after the model inference.

[0084] (4) Model serialization: Converting the model into a format suitable for inference, which involves converting the model into a format specific to the hardware in the middleware.

[0085] (5) Load the model binding container: Load the serialized model into a container, which may be used to manage the lifecycle and resources of the model.

[0086] (6) Model execution: Run the model and perform inference calculations.

[0087] (7) Copy the output data back to the host memory: Copy the results of the model inference from the model execution environment back to the host memory for further processing or use.

[0088] The runtime middleware technology of this invention will provide Cambricon's intelligent processor platform with an efficient, flexible, and easy-to-use operating environment. This will not only accelerate the research and development of artificial intelligence technology but also promote the rapid innovation and deployment of related applications. In this way, we hope to contribute to technological advancements and industrial development in the field of artificial intelligence, enabling Cambricon's intelligent processor platform to realize its full potential in the era of artificial intelligence.

[0089] This invention is applied to the Cambricon MLU370 intelligent processor platform, which can support the deployment of ONNX models on the MLU370 accelerator card, realize the compilation and running of deep learning models, and form a runtime middleware that supports the Cambricon intelligent processor platform. In the integration of the Cambricon intelligent processor hardware platform and ONNXRuntime, the middleware ensures that the two can interact seamlessly and realize the effective transmission of data and control flow.

[0090] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0091] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a middleware running method on a processor platform.

[0092] In one embodiment of the present invention, a storage medium is also provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the operating system of the terminal. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the middleware operation method for a processor platform in the above embodiments.

[0093] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0094] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A middleware execution method for a processor platform, characterized in that, include: Get the input data; Memory serialization of the ONNX model based on the input data; The ONNX model, after being serialized in memory, is decomposed into multiple subgraphs using a model segmenter. Multiple subgraphs are assigned to different model executors via a Provider registrar; The model executor is responsible for interacting with the execution provider to execute the various parts of the model on the appropriate hardware; Among them, the execution providers include CAMBRICONExecutionProvider, which supports the Cambricon hardware platform; Construct an ONNXRuntime::cambricon::CambriconModelPreparation object, construct the input map (input_map) and output map (output_map), and map the input and output names of the nodes to their indices; The model executor runs the model, performs inference calculations, and generates processed output data. The model executor runs the model and performs inference calculations, including: To perform model inference, the cnrtQueueCreate function is used to create a Cambrian runtime queue_ to manage the queue for model execution. Call the Enqueue method of the context object in CambriconModelPreparation, and pass the input prepare.inputSet_, the output prepare.outputSet_, and the queue_ to the Enqueue method to schedule the actual execution of the model; The processed output data is output from the system.

2. The middleware operation method for a processor platform according to claim 1, characterized in that, Before performing memory serialization of the ONNX model based on the input data, the process also includes: Determine whether the current model file in the input data has been compiled. If not, compile the ONNX model file.

3. The middleware operation method for a processor platform according to claim 2, characterized in that, The compilation of the ONNX model file includes: Construct a Graph class, and use the Status ParserONNXModel function to convert the serialized_string format *.onnx model into a Network structure, and then construct a me::Graph object from the ONNX model.

4. The middleware operation method for a processor platform according to claim 2, characterized in that, If the current model file in the input data has already been compiled, the compiled file "*.mm" is read into memory through the ModelLoadFromFile function, a me::ModelDesc object is constructed, the number of input and output data is set, and the compiled model in memory is bound to the model ID through me::ModelDescContainer. Subsequently, memory serialization is performed directly.

5. The middleware operation method for a processor platform according to claim 1, characterized in that, The in-memory serialization of the ONNX model based on input data includes: Configure the builder, using the me::ModelBufferData object to store the serialized model data and size in memory, named "*.mm"; manage it through the me::ModelDescContainer object, bind it to the model ID, and convert the model into a format suitable for inference.

6. The middleware operation method for a processor platform according to claim 1, characterized in that, After obtaining the input data, the input data is preprocessed to meet the input requirements of the model.

7. A middleware runtime system for a processor platform, characterized in that, include: The data acquisition module is used to acquire input data; The ONNX model module is used for in-memory serialization of ONNX models based on input data; The model segmentation module is used to decompose the memory-serialized ONNX model into multiple subgraphs using a model segmenter. Multiple subgraphs are assigned to different model executors via a Provider registrar; The model executor is responsible for interacting with the execution provider to execute the various parts of the model on the appropriate hardware; Among them, the execution providers include CAMBRICONExecutionProvider, which supports the Cambricon hardware platform; Construct an ONNXRuntime::cambricon::CambriconModelPreparation object, construct the input map (input_map) and output map (output_map), and map the input and output names of the nodes to their indices; The model execution module is used to run the model, perform inference calculations, generate processed output data, and output the processed output data from the system. The model executor runs the model and performs inference calculations, including: To perform model inference, the cnrtQueueCreate function is used to create a Cambrian runtime queue_ to manage the queue for model execution. Call the Enqueue method of the context object in CambriconModelPreparation, passing the input prepare.inputSet_, the output prepare.outputSet_, and the queue_ to the Enqueue method to schedule the actual execution of the model.

8. A computer device, characterized in that, include: A processor is used to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, performs a middleware execution method for a processor platform as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a middleware running method for a processor platform as described in any one of claims 1-6.

Images