Method and device for running open source engine, computer device and storage medium

Abstract

This application relates to the field of computer technology, specifically to a method, apparatus, computer device, and storage medium for running an open-source engine. The method for running the open-source engine is applied to a hardware extension plugin system for an inference engine. The hardware extension system includes at least an inference engine module and a hardware extension plugin module. The method includes: determining whether the current hardware is configured with all target operators required for inference computation of the model to be computed; if missing, determining whether the inference engine interfaced with the current hardware is configured with missing hardware operators; if the inference engine interfaced with the current hardware is not configured with missing hardware operators, calling the missing hardware operators from a pre-configured CPU operator library in the CPU through the hardware extension plugin system; and registering all target operators to the inference engine and / or the current hardware. This provides a secure, reliable method for third-party hardware to access an open-source engine with complete evolvability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, computer device, and storage medium for running an open-source engine. Background Technology

[0002] The general-purpose inference engine is a unified, adaptive, hardware-aware software system for loading and running AI (Artificial Intelligence) models with various architectures. It automatically adapts to various hardware platforms (such as from cloud GPUs to edge device CPUs), dynamically optimizes computation and memory usage at runtime, and provides a standardized inference interface for upper-layer applications.

[0003] Existing technologies have explored hardware extensions of inference engine frameworks in terms of interface abstraction, operator integration, and runtime binding, but they all face one or more key problems, such as high integration coupling, lack of automatic rollback capability, inconsistent support for emerging formats, and insufficient runtime security and observability.

[0004] Therefore, there is an urgent need for a third-party hardware plug-in access solution that provides a secure, reliable, and fully evolvable solution for general-purpose inference engines. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and storage medium for running an open-source engine.

[0006] A first aspect of this application provides a method for running an open-source engine, applied to a hardware extension plugin system for the open-source engine. The hardware extension system includes at least: an inference engine module and a hardware extension plugin module; the inference engine module is configured with a model to be computed and an inference engine; the hardware extension plugin module includes at least: a plugin definition module and a CPU rollback operator module; the method includes: Determine whether the current hardware contains all the target operators required for inference calculations of the model to be computed. If missing, it is determined whether the inference engine interfaced with the current hardware is configured with a hardware missing operator; wherein, the hardware missing operator refers to the CPU operator in the target operator that is not configured in the current hardware; If the inference engine that interfaces with the current hardware is not configured with the missing hardware operator, the missing hardware operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system. Register all the target operators to the inference engine and / or the current hardware.

[0007] In an optional embodiment of this application, after determining whether the inference engine interfaced with the current hardware is configured with a hardware missing operator, the method further includes: If the inference engine that interfaces with the current hardware is configured with the hardware missing operator, then determine whether the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU. If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator configured in the inference engine and other target operators in the current hardware are called through the hardware extension plug-in system to perform inference computation on the model to be computed.

[0008] In an optional embodiment of this application, after determining whether the inference engine interfaced with the current hardware is configured with a hardware missing operator, the method further includes: If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU-compatible algorithm, the hardware missing operator configured in the inference engine and other target operators in the current hardware are invoked through the hardware extension plugin system to perform inference calculations on the model to be calculated.

[0009] In an optional embodiment of this application, if the inference engine interfaced with the current hardware is not configured with a hardware missing operator, then calling the hardware missing operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system includes: If the inference engine that interfaces with the current hardware is not configured with the missing hardware operator, then determine whether the data type of the computation required by the model to be computed is in the preset set of data types of computation supported by the CPU. If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0010] In an optional embodiment of this application, the step of calling the missing hardware operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system if the inference engine interfaced with the current hardware is not configured with a hardware missing operator further includes: If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU compatibility, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0011] In an optional embodiment of this application, the plug-in definition module includes: a plug-in interface layer providing standardized API interfaces, a hardware abstraction layer, and a hardware driver interface layer; wherein, the method further includes: The configuration information required for the inference engine to perform inference calculations is obtained through the plugin interface layer; the configuration information includes at least: the target operator required by the model to be calculated and the data type of the calculation. The hardware abstraction layer executes the interface functions corresponding to each of the standardized API interfaces; the interface functions refer to the task functions executed by the inference engine when performing inference calculations. The hardware driver interface layer encapsulates the usage of peripherals and provides the CPU with an interface to call the standard display driver model.

[0012] In one optional embodiment of this application, the standardized API interface includes at least one of the following: an interface for registering hardware-supported operators, an interface for allocating hardware memory, a host-to-device data transfer interface, an interface for executing hardware operators, a weight pre-download interface, a missing operator list interface, and a weight mode setting interface.

[0013] In one optional embodiment of this application, the hardware abstraction layer includes at least one of the following: an operator mapper, a memory manager, a data synchronizer, and a rollback checker.

[0014] In an optional embodiment of this application, registering all the target operators to the inference engine and / or the current hardware includes: The weight list of the current weight mode of the model to be calculated can be obtained through the weight mode setting interface; Obtain and register all target operators by registering the hardware-supported operator interface, and receive the list of hardware-supported operators and the list of hardware-unsupported operators fed back by the CPU. If the target operator is in the list of operators supported by the hardware, the weights in the weight list are pre-downloaded to the hardware driver interface layer in the plug-in definition module through the weight pre-download interface. The memory of the current device is allocated through the hardware memory allocation interface, and the weight list is stored.

[0015] In an optional embodiment of this application, registering all the target operators to the inference engine and / or the current hardware further includes: The missing operator list is obtained through the missing operator list interface; wherein, the missing operator list contains all the hardware missing operators; The CPU rollback operator module is invoked to roll back to the CPU for inference calculation by registering all target operators with CPU operators; wherein, the list of all target operators includes the list of missing operators.

[0016] In an optional embodiment of this application, the hardware extension plug-in module further includes: a model weight module; the method further includes end-to-end performance testing of the model to be computed; the testing process includes: The model to be computed is subjected to single-layer transformer weight processing; The performance of the model to be computed is tested based on the weight processing of the single-layer transformer.

[0017] In an optional embodiment of this application, the hardware extension plug-in module further includes: a model weight module; the method further includes end-to-end performance testing of the model to be computed; the testing process includes: The model to be computed is subjected to weight processing using a two-layer transformer, and / or expert number segmentation processing. Based on the weight processing of the two-layer transformer and / or the expert number segmentation processing, the parallelism of the model to be computed is tested.

[0018] A second aspect of this application provides an apparatus for running an open-source engine, characterized by a hardware extension plugin system applied to the open-source engine, the hardware extension system comprising at least: an inference engine module and a hardware extension plugin module; the inference engine module is configured with a model to be computed and an inference engine; the hardware extension plugin module comprises at least: a plugin definition module and a CPU rollback operator module; the apparatus comprises: The first judgment module is used to determine whether the current hardware is configured with all the target operators required for the inference calculation of the model to be calculated. The second judgment module is used to determine whether the inference engine interfaced with the current hardware is configured with a hardware missing operator if the missing operator is missing; wherein, the hardware missing operator refers to the CPU operator among the target operators that are not configured in the current hardware. The calling module is used to call the hardware missing operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system if the inference engine interfaced with the current hardware is not configured with the hardware missing operator. A registration module is used to register all the target operators to the inference engine and / or the current hardware.

[0019] A third aspect of this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0020] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0021] The open-source engine operation method provided in this application adopts a CPU backup computing mechanism. A complete AI large-scale model CPU operator library is built into the CPU, meaning all target operators for the model to be computed are configured in the CPU. If a target operator is missing in the current hardware, it is determined whether the inference engine interfaced with the current hardware has the missing hardware operator configured. If it does, the operator already implemented or configured in the inference engine is called for inference computation. If the inference engine interfaced with the current hardware does not have the missing hardware operator configured, the missing hardware operator is called from the pre-configured CPU operator library in the CPU through the hardware extension plug-in system. That is, when a hardware operator is missing, the computation automatically reverts to CPU computation. This ensures a complete CPU operator library, and after computation, the computation can revert to the current hardware to output the inference result, achieving end-to-end AI large-scale model inference. This allows any third-party hardware to smoothly access the inference engine and securely, reliably, and completely realize the inference evolution of the model to be computed.

[0022] The open-source engine operation method provided in this application is applied to a hardware extension plug-in system for an inference engine. The hardware extension system includes at least an inference engine module and a hardware extension plug-in module. The hardware extension plug-in module includes at least a plug-in definition module and a CPU rollback operator module. Through the hardware extension plug-in access mechanism implemented by each configured module, any hardware device can interface with a general inference engine and automatically register and replace implemented operators in the inference engine. It safely rolls back missing hardware operators that cannot be implemented and registers all target operators to the inference engine and / or the current hardware, thereby achieving end-to-end availability, verifiable performance, and evolvable integration for large-scale AI model inference. In summary, this application provides a secure, reliable, and fully evolvable method for third-party hardware to access an open-source engine. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of a hardware extension plug-in system architecture for an inference engine provided in one embodiment of this application; Figure 2 A flowchart illustrating the operation method of an open-source engine provided in one embodiment of this application; Figure 3 A flowchart illustrating the operation method of an open-source engine provided in one embodiment of this application; Figure 4 A flowchart illustrating the operation method of an open-source engine provided in one embodiment of this application; Figure 5 A flowchart illustrating the operation method of an open-source engine provided in one embodiment of this application; Figure 6 A flowchart illustrating the operation method of an open-source engine provided in one embodiment of this application; Figure 7 A schematic diagram of the operating device structure of an open-source engine provided in one embodiment of this application; Figure 8 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0024] In the process of developing this application, the inventors discovered that there is an urgent need for a third-party hardware plug-in access solution that provides a secure, reliable, and fully evolvable solution for general-purpose inference engines.

[0025] To address the aforementioned issues, this application provides a method, apparatus, computer device, and storage medium for running an open-source engine.

[0026] The solutions in this application embodiment can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0027] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0028] The following is a brief description of the application environment for the open-source engine running method provided in the embodiments of this application: Please see Figure 1The open-source engine operation method provided in this application embodiment is applied to the hardware extension plugin system of the inference engine. The hardware extension system of the inference engine includes at least: an inference engine module and a hardware extension plugin module.

[0029] The inference engine module is configured with a model to be computed and an inference engine. The model to be computed refers to the large AI model that performs inference computation, such as the GPT series, Llama series, Qwen series, DeepSeek series, etc., which are not exhaustive and can be flexibly selected or set according to the actual situation. The general inference engine (such as vLLM) serves as the upper layer of the hardware extension plugin system of the inference engine, responsible for things like standard model loading, computation graph parsing, task scheduling, and inference execution. The model to be computed is used to perform inference computation, which requires a large number of operators, which are generally provided by GPUs and CPUs. During the execution of the model to be computed, third-party hardware manufacturers may want to use their own hardware to completely replace GPUs for large model inference. In this case, the hardware manufacturers will need to implement at least all the target operators required by the model to be computed on their own hardware. If any operator is missing, the large model inference cannot be executed. The open-source engine operation method provided in this application embodiment is designed to complete inference computation even in the case of missing operators.

[0030] The hardware extension plug-in modules include: plug-in definition module, model weight module, and CPU backoff operator module, etc., which will not be listed exhaustively here.

[0031] The plugin interface layer in the plugin definition module provides a standardized API interface for interaction between external hardware and the CPU. For the hardware extension plugin system, this standardized API interface connects the external hardware to the inference engine, enabling access to the CPU even when operators are missing in the external hardware. The CPU operators then continue the inference computation of the model to be computed, achieving end-to-end (CPU to external hardware device) complete inference computation. For external hardware devices, there is no need to develop a general inference framework. They can access the inference engine module through the standardized API interface in this embodiment, enabling interaction with the general engine within the inference engine module and accelerating the computation of large AI models configured in the inference engine module. Even in the early stages of hardware operator development and inference engine integration, or when hardware resources are insufficient, additional hardware devices can be connected through the standardized API interface to achieve end-to-end (CPU to external hardware device) inference computation.

[0032] The model weight module is used to analyze the original large AI model and build a model containing only single / double layer transformer weights. This allows the new model to contain all the computations and logic required by the original model, but with very low computation and storage requirements. This allows for quick verification of whether the external hardware supports the computational functions required by the model being computed.

[0033] The CPU rollback operator module is used to convert the data type of the data to be calculated into a data type supported by the CPU when the CPU does not support certain data type calculations. After the calculation is complete, the corresponding operator is rolled back to the CPU or hardware device via the CPU rollback operator module. The CPU rollback operator module is also used to register a CPU-implemented operator when the connected inference engine module does not have a CPU version that implements the operator required by the model to be calculated. This CPU-implemented operator, for input data types that do not support CPU calculations (such as fp8), converts the data type to a data type supported by the CPU. After the calculation is complete, the output data type is modified back to the original data type and rolled back to the corresponding hardware device.

[0034] In the early stages of developing the general inference engine, the hardware only implemented the registration function of a few operators. However, it was necessary to verify whether the hardware device supported the complete end-to-end functionality to run a large AI model. In this embodiment, a CPU rollback operator module is set up to call the required operators that are not currently registered on the hardware from other hardware devices. After use, the CPU rollback operator module is used to roll back the operators, thereby completing the verification of whether the hardware device supports running a large AI model.

[0035] Currently, mainstream large-scale AI models tend to have increasingly more model parameters. To quickly obtain inference results, low-precision computation and weight quantization are gradually replacing high-precision computation. Since CPUs do not natively support low-precision computations such as FP8 (E4M3 / E5M2) and FP4, this application embodiment processes the weights of mainstream large-scale AI models, for example, by adding weights for low-precision data types such as float16, bfloat16, and float32. For computation types that are not supported by CPUs, such as FP8 and W8A8, the data that is not supported by CPUs is first converted into high-precision computation data such as float16, bfloat16, and float32 that are supported by CPUs. Then, it is determined whether the inference engine has implemented high-precision CPU computation operators. If it does, the CPU operators implemented by the inference engine are directly called; if not, the pre-registered CPU operators completed in this application embodiment are used, thereby achieving a complete CPU fallback function while making full use of the general inference engine.

[0036] Please continue reading Figure 1 In an optional embodiment of this application, the plug-in definition module includes: a plug-in interface layer, a hardware abstraction layer, and a hardware driver interface layer, wherein: The plugin interface layer provides a standardized API interface for obtaining configuration information required for engine inference computation. In one optional embodiment of this application, the standardized API interface is a C++ API, which may include at least one of the following: a hardware-supported operator registration interface (register_ops()), a hardware memory allocation interface (allocate_memory()), a host-to-device data transfer interface (copy_data_to_device()), a hardware operator execution interface (execute_ops()), a weight pre-download interface (download_weights()), a missing operator list interface (get_missing_ops()), and a weight mode setting interface (set_weight_mode(), including single / double / complete layers). For example, the missing operator list interface (get_missing_ops()) can automatically generate a missing operator list to provide clear priorities for development and avoid blind development; at the same time, it provides a single-layer verification mode to simplify debugging complexity and quickly locate problems. The system dynamically maintains an operator registry, provides a list of missing operators, guides hardware development priorities, and in the early stages of development, performs calculations by reverting to the CPU for operators not supported by the hardware. The calculation results are then organized and analyzed to summarize the set of operators currently used by the hardware and the set of operators used by the CPU, and sorted by frequency of use to provide development priorities for hardware development.

[0037] The hardware abstraction layer is used to execute the interface functions corresponding to each of the standardized API interfaces; Taking the standardized API interface examples described above as examples, when using hardware for AI large-scale model inference calculations, the corresponding interface functions of each standardized API interface include: obtaining the required weight list by setting the weight mode (single / dual / full layer) using `set_weight_mode()`; after registering the operator using the hardware-supported operator registration interface `register_ops()`, a list of operators supported by the hardware will be generated; and obtaining the missing operator list using the missing operator list get_missing_ops() interface, with the missing parts calculated using the CPU rollback operator module. For hardware-supported operators, weights are pre-downloaded to the hardware using the weight pre-download interface `download_weights()`, and hardware memory is allocated using the hardware memory allocation interface `allocate_memory()` to store the required weights. For operators not supported by the hardware, weights are pre-downloaded to the CPU using the weight pre-download interface `download_weights()`.

[0038] In one optional embodiment of this application, the hardware abstraction layer is used to implement the specific functions of the interface layer, and the hardware abstraction layer may include, for example, such as Figure 1 In the middle: at least one of the following: operator mapper, memory manager, data synchronizer, and rollback checker. Among them: The operator mapper is used to construct a mapping relationship between general operator names and hardware-specific implementations. For example, the MatMul operator used in the model to be computed is named MatMul at the framework layer. However, the operator names are not consistent in the actual computation on different hardware. The operator mapper ensures that when MatMul is called at the framework layer, it points to the specific computation operator used by the hardware. The memory manager is used to uniformly manage the allocation and release of hardware memory. For example, when an operator is being computed, memory is allocated for the output result, and after the computation, the memory for the input data is released. The data synchronizer is used to ensure data consistency between the CPU and the hardware used. During AI large-scale model inference computation, even if it is not necessary to roll back to the CPU operator, the CPU is still needed for scheduling and auxiliary computation, so it is necessary to ensure data consistency between the CPU and the hardware used. The rollback judge is used to intelligently determine the operator execution path. When an operator has not achieved a mapping with the hardware, it rolls back to the CPU for computation.

[0039] The hardware driver interface layer encapsulates the usage of hardware devices and provides the CPU with an interface to call standard display driver models. The driver interface layer encapsulates the complex details of using hardware devices and provides the CPU with an interface to call standard display driver models. It abstracts and encapsulates the complexity of the underlying hardware, providing a unified, concise, and stable interface for upper-layer software (such as AI frameworks and applications). These "complex details" can include, for example, hardware register and memory mapping operations, heterogeneous computing resource management, memory and data movement, interrupt and event handling, error handling and recovery, multi-device and virtualization, performance optimization and tuning, standardized interface adaptation, firmware and microcode management, etc. These are not exhaustive and can be flexibly configured according to actual needs.

[0040] Please see Figure 2 The following embodiments use the aforementioned hardware extension plugin system as the execution entity, applying the open-source engine operation method provided in this application to the aforementioned inference engine hardware extension plugin system to specifically illustrate the inference calculation of the model to be calculated. The open-source engine operation method provided in this application includes the following steps 201-204: Step 201: Determine whether the current hardware is configured with all the target operators required for the inference calculation of the model to be calculated; The current hardware refers to third-party hardware developed by the developer or used by the user and requiring access to the inference engine. The model to be computed refers to the large AI model used to perform inference computation. As mentioned above, the large AI model can be, for example, the GPT series, Llama series, Qwen series, DeepSeek series, etc., and is not exhaustively listed here. It can be flexibly selected or set according to the actual situation. The target operator is all the operators required by the model to be computed to perform inference computation. The operator can be completely provided by the third-party hardware. If the third-party hardware cannot provide it completely, it can be implemented according to the running method of the open source engine provided in the embodiment of this application. For example, the target operator is queried in the inference engine module through the following steps; since the CPU is generally configured with all or most of the necessary operators of the large AI model, in this embodiment of the application, if the inference engine module cannot provide it completely, the CPU fallback method is used to fall back to the CPU for inference computation, so as to realize the whole process of inference computation of the large AI model.

[0041] Step 202: If missing, determine whether the inference engine connected to the current hardware is configured with a hardware missing operator; The missing hardware operators refer to CPU operators among the target operators that are not configured in the current hardware; that is, some of the target operators necessary for the execution of the computation model are configured in the current hardware, while others need to be found in the inference engine or CPU.

[0042] Step 203: If the inference engine interfaced with the current hardware is not configured with the hardware missing operator, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system. The CPU operator library in the CPU stores all the target operators required when the model to be computed performs inference computation.

[0043] Step 204: Register all target operators to the inference engine and / or the current hardware.

[0044] The missing hardware operators and existing hardware operators are registered to the inference engine and / or the current hardware for backend inference calculation. In this way, the operators in the current hardware are continuously updated, forming positive feedback during use. Through continuous iteration and updates during use, the current device can eventually load all target operators of all models to be calculated.

[0045] The core of mainstream large-scale AI models based on Transformers lies in repeatedly stacking "a few types of operator combinations," primarily matrix multiplication, attention, normalization, activation, and some simple element / index operations. These operators include, but are not limited to: Linear, MatMul, Batched MatMul, Softmax, Mul, mask, LayerNorm, RMSNorm, GLU, SwiGLU, Embedding, Argmax, TopK / TopP, etc. While these operators are limited, the limited computing power of CPUs and their lack of support for INT4 / FP8 data types mean that even with a large number of CPU operators available in current inference engines, complete end-to-end computation for most large-scale AI models is still impossible.

[0046] Taking a general-purpose inference engine primarily designed for GPUs / CPUs as an example, the development of large AI models is increasingly trending towards large-scale parameters, and inference computation is trending towards low-precision, high-speed computation. Using a pure CPU on a general-purpose inference engine cannot successfully run inference for large AI models with low-precision computations such as FP8. Running a large AI model like Deepseek_v3 on an inference engine vLLM using a pure CPU will show that the computation fails because the CPU does not support FP8 computation. This application's embodiment uses a CPU fallback mechanism, enabling the successful execution of inference computations like Deepseek_v3 on an inference engine vLLM using only the CPU.

[0047] The open-source engine operation method provided in this application adopts a CPU backup computing mechanism. A complete AI large-scale model CPU operator library is built into the CPU, meaning all target operators for the model to be computed are configured in the CPU. If a target operator is missing in the current hardware, it is determined whether the inference engine interfaced with the current hardware has the missing hardware operator configured. If it does, the operator already implemented or configured in the inference engine is called for inference computation. If the inference engine interfaced with the current hardware does not have the missing hardware operator configured, the missing hardware operator is called from the pre-configured CPU operator library in the CPU through the hardware extension plug-in system. That is, when a hardware operator is missing, the computation automatically reverts to CPU computation. This ensures a complete CPU operator library for the model to be computed. After computation, the computation can revert to the current hardware to output the inference result, achieving end-to-end AI large-scale model inference. This allows any third-party hardware to smoothly access the inference engine and securely, reliably, and completely realize the inference evolution of the model to be computed.

[0048] The open-source engine operation method provided in this application is applied to a hardware extension plug-in system for an inference engine. The hardware extension system includes at least an inference engine module and a hardware extension plug-in module. The hardware extension plug-in module includes at least a plug-in definition module and a CPU rollback operator module. Through the hardware extension plug-in access mechanism implemented by each configured module, any hardware device can interface with a general inference engine and automatically register and replace implemented operators in the inference engine. It safely rolls back missing hardware operators that cannot be implemented and registers all target operators to the inference engine and / or the current hardware, thereby achieving end-to-end availability, verifiable performance, and evolvable integration for large-scale AI model inference. In summary, this application provides a secure, reliable, and fully evolvable method for third-party hardware to access an open-source engine.

[0049] In one optional embodiment of this application, after step 202 above, which involves determining whether the inference engine interfaced with the current hardware is configured with a hardware missing operator, the following two scenarios are included: The first scenario: If the inference engine interfaced with the current hardware is configured with the hardware missing operator, then the following steps are included: Determine whether the data type required by the model to be computed is in the preset set of data types supported by the CPU; If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator configured in the inference engine and other target operators in the current hardware are called through the hardware extension plug-in system to perform inference computation on the model to be computed.

[0050] If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU-compatible algorithm, the hardware missing operator configured in the inference engine and other target operators in the current hardware are invoked through the hardware extension plugin system to perform inference calculations on the model to be calculated.

[0051] It should be explained that among the data types supported by the CPU, some only support data storage of that specific data type, while others support both data storage and data computation. This application's embodiment requires using the CPU to perform computation, and therefore requires data types that the CPU supports for computation.

[0052] In the second scenario, step 203 above, where the inference engine interfaced with the current hardware is not configured with a hardware missing operator, involves calling the hardware missing operator from the pre-configured CPU operator library in the CPU through the hardware extension plugin system. Calling the hardware missing operator from the pre-configured CPU operator library in the CPU through the hardware extension plugin system includes the following steps: Determine whether the data type required by the model to be computed is in the preset set of data types supported by the CPU: If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0053] If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU compatibility, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0054] The preset set of CPU-supported computational data types includes all data types supported by the CPU, such as FP32; conversely, the set of CPU-unsupported computational data types includes all data types not supported by the CPU, such as INT4 / FP8. In this embodiment, if the computational data type required by the model to be computed is not in the preset set of CPU-supported computational data types, it is converted to any data type in the set of CPU-supported computational data types, thus obtaining data compatible with the CPU. After computation, the result is rolled back to the original data type. This ensures that end-to-end large-scale AI model inference can be achieved during the initial integration stage of any hardware by using a rollback approach to complete CPU computation.

[0055] In this application embodiment, data whose data type is not in the preset set of CPU-supported computing data types is converted into CPU-compatible computing data before inference calculation. By unifying the quantization format of each interface, observability and verifiability, runtime isolation, and version compatibility with different hardware devices are achieved. Without relying on hardware devices such as GPUs, third-party hardware devices can be used for end-to-end inference of large AI models quickly and stably, and quantifiable performance benefits and engineering maintainability can be obtained, greatly improving the compatibility of third-party hardware device access.

[0056] Please see Figure 3 In an optional embodiment of this application, the plugin definition module includes: a plugin interface layer, a hardware abstraction layer, and a hardware driver interface layer that provide standardized API interfaces; the specific structure and functions of the plugin interface layer, the hardware abstraction layer, and the hardware driver interface layer have been described in detail in the above embodiments and will not be repeated here. Correspondingly, the above-mentioned open-source engine operation method further includes the following steps 301-303: Step 301: Obtain the configuration information required by the inference engine for inference calculation through the plug-in interface layer; The configuration information includes at least the target operators required by the model to be computed and the data type of the computation; this configuration information is used to realize intelligent management of operators, for example, it can dynamically maintain an operator registry, provide a list of missing operators, and guide hardware development priorities.

[0057] Step 302: Execute the corresponding interface functions of each of the standardized API interfaces through the hardware abstraction layer; The interface function refers to the task function executed by the inference engine when performing inference calculations; the specific interface categories and functions have been described in detail in the above embodiments and will not be repeated here.

[0058] Step 303: Encapsulate the usage of peripherals through the hardware driver interface layer and provide the CPU with an interface to call the standard display driver model.

[0059] Encapsulating peripheral hardware details includes, but is not limited to: register operations, timing control, and interrupt handling, to provide a unified hardware access interface and facilitate porting and testing.

[0060] This application embodiment obtains the configuration information required by the inference engine to perform inference calculations through the plug-in interface layer, executes the corresponding interface functions of each standardized API interface through the hardware abstraction layer, encapsulates the usage content of peripherals through the hardware driver interface layer, and provides the CPU with an interface to call the standard display driver model, thereby realizing hardware support for the operation of the open source engine in this application embodiment.

[0061] Please see Figure 4 In an optional embodiment of this application, step 204 above, the step of registering all the target operators to the inference engine and / or the current hardware, includes the following steps 401-404: Step 401: Obtain the weight list of the current weight mode of the model to be calculated through the weight mode setting interface; Step 402: Obtain and register all target operators through the registered hardware-supported operator interface, and receive the list of hardware-supported operators and the list of hardware-unsupported operators fed back by the CPU. Step 403: If the target operator is in the list of operators supported by the hardware, then the weights in the weight list are pre-downloaded to the hardware driver interface layer in the plug-in definition module through the weight pre-download interface. Step 404: Allocate memory for the current device through the hardware memory allocation interface and store the weight list.

[0062] Using any third-party hardware, the system implements the registration function of a single operator through the aforementioned plugin definition module. After completing the plugin definition, the deepseek_v3 inference computation is run again. All target operators are obtained and registered through the hardware-supported operator interface. The system receives a list of hardware-supported operators and a list of hardware-unsupported operators from the CPU. Registered operators automatically run on the hardware being used, while unregistered operators run on the CPU. A consolidated list of operators required for subsequent hardware registration is provided, i.e., the list of hardware-supported operators and the list of hardware-unsupported operators. The CPU rollback mechanism ensures that the system can still operate normally in the event of hardware malfunction or missing operators, supporting a smooth transition from development to production. It maintains interface consistency, executing corresponding functions through corresponding interfaces. A unified abstraction layer supports rapid integration of various types of hardware accelerators, enabling end-to-end integration verification even when operator implementations are incomplete, significantly shortening the development cycle.

[0063] Please see Figure 5In an optional embodiment of this application, step 204 above, the step of registering all the target operators to the inference engine and / or the current hardware, further includes the following steps 501-502: Step 501: Obtain the list of missing operators through the missing operator list interface; The missing operator list includes all of the hardware missing operators; Step 502: Call the CPU rollback operator module to roll back the list of all target operators to the CPU for inference calculation using the CPU operator registration method; The list of all target operators includes the list of missing operators. The missing CPU operators and existing hardware operators are registered to the computing backend for inference computation using an operator registration method.

[0064] In this embodiment, all target operators required by the model to be computed can be registered through operator registration. If the inference engine does not contain all the required target operators, the calling method of the operators when the model to be computed is executed is determined in the above manner. For example, in an optional embodiment of this application, the hardware to be used for execution can be defined by setting the device. If device=CPU, the CPU operators are used for computation. If device=any other hardware, when the operator is actually called, it is first determined whether the required operator exists on this hardware. If it does not exist, it falls back to the CPU for inference computation. If it exists, the hardware operator is called.

[0065] Since most general-purpose large AI models (such as the GPT series, Llama series, Qwen series, etc.) are essentially decoder-only Transformer architectures, the structural changes of these large AI models are not significant, with more minor adjustments to details (normalization position, activation function, RoPE details, KV cache organization, etc.). Many new-generation large AI models (such as Gemini 1.5Pro, DeepSeek-V3 / R1, and the subsequent Llama 4 series, etc.) have shifted to sparse MoE Transformer architectures. These large AI models use a large number of "expert subnetworks" in some layers, and each token (the core processing unit of the large language model) only activates a few experts (conditional computation), achieving a model architecture with "huge total parameters and controllable activation parameters for each token". They also improve inference speed through quantization methods such as INT4 / FP8, making it impossible for CPUs to complete end-to-end AI large model inference computation. In this embodiment of the application, when performing end-to-end computation for large-scale AI model inference, if the hardware does not register the operator, computation can be performed by rolling back the CPU. This embodiment determines the data type at the time of computation. If it is a computation type supported by the CPU, the CPU is used directly for computation. If it is a computation type not supported by the CPU, the data is first modified to a CPU-compatible data type, and the corresponding data type weight is selected before inference computation. After the CPU computation is completed, the data type is converted back to the original data type that the operator should have obtained. This allows for end-to-end computation and verification of large-scale AI model inference using CPU rollback, whether in the early stages of inference engine development or when hardware resources are insufficient. Furthermore, the CPU rollback function also provides operator function verification for hardware operator development. After the hardware implements the registration of a certain operator, to determine whether the operator's computation result is correct, the CPU rollback method in this embodiment can be used to obtain the corresponding accurate result, greatly saving the development and verification cycle.

[0066] Please see Figure 6 In an optional embodiment of this application, the hardware extension plugin module further includes: a model weight module; the above-mentioned open-source engine operation method further includes performing end-to-end performance testing on the model to be calculated; the testing process includes the following steps 601-604: Step 601: Perform single-layer transformer weight processing on the model to be calculated; Step 602: Perform performance testing on the model to be calculated based on the weight processing of the single-layer transformer.

[0067] The current trend in the development of large AI models is towards increasingly larger parameter scales. This is due to both the increased number of model layers and the greater computational dimensionality within each layer. Implementing end-to-end inference for a large AI model on new hardware often requires downloading a large number of weights and performing extensive computations. During end-to-end inference, if a problem occurs at a certain node, it becomes difficult to pinpoint the specific location and operator type. Resolving the issue and re-running the model then consumes significant time, slowing down development progress. For large language models with a transformer architecture, this application's embodiment uses single-layer transformer weight processing, retaining only the weights of one transformer layer, significantly reducing memory usage. In the early stages of development, for each hardware, only the support and performance of the required operators can be verified. Ensuring data parallelism and tensor parallelism for the large AI model allows for end-to-end inference, significantly reducing hardware memory requirements and debugging complexity. After verifying that the required operators are fully implemented on any hardware, complete end-to-end inference computation is then performed using the full weights, thereby improving overall efficiency.

[0068] Step 603: Perform weight processing on the model to be calculated using a two-layer transformer, and / or perform expert number segmentation processing. Step 604: Based on the weight processing of the two-layer transformer and / or the expert number segmentation processing, perform parallelism testing on the model to be computed.

[0069] For large language models with a transformer architecture, this application's embodiment uses a two-layer transformer weight processing method, retaining only the weights of two transformer layers, thus significantly reducing memory usage. In the early stages of development, it allows for verification of the support and performance of only the required operators on various hardware devices, ensuring pipelined parallelism for the large AI model. This enables end-to-end inference for the large AI model, significantly reducing hardware memory requirements and debugging complexity. After verifying that the required operators are fully implemented on any hardware, complete end-to-end inference computation is then performed using the full weights, thereby improving overall efficiency.

[0070] The expert number partitioning in the MoE Transformer architecture ensures expert parallelism for large AI models. For sparse MoE Transformer architectures, this application's embodiments utilize expert number partitioning to minimize storage weights while maintaining the MoE Transformer architecture, thereby reducing memory consumption and improving inference computation efficiency.

[0071] This application's embodiments, through weight processing methods involving single-layer transformers, two-layer transformers, and expert number segmentation, can ensure support for the computational characteristics of a complete large AI model while simultaneously verifying hardware performance early on, providing early judgment for hardware development. In an optional embodiment of this application, the option to use complete weights can also be retained, allowing for complete end-to-end inference computation by invoking the complete weights from each layer of the large AI model when the hardware resources of the general inference engine are sufficient.

[0072] The embodiments of this application provide multiple weight selection options to achieve unified inference engine integration, enabling small and medium-sized hardware manufacturers to quickly access the mainstream inference ecosystem. Through a large number of CPU rollback operators, it supports the incremental functional improvement of hardware manufacturers, allowing them to get started easily and conduct early verification, achieving the lowest risk and lowest cost investment.

[0073] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0074] Please see Figure 7 One embodiment of this application provides a running device 700 for an open-source engine, applied to a hardware extension plugin system for the open-source engine. The hardware extension system includes at least: an inference engine module and a hardware extension plugin module; the inference engine module is configured with a model to be computed and an inference engine; the hardware extension plugin module includes at least: a plugin definition module and a CPU rollback operator module; the device 700 includes: The first judgment module 710 is used to determine whether the current hardware is configured with all the target operators required for the inference calculation of the model to be calculated. The second judgment module 720 is used to determine whether the inference engine connected to the current hardware is configured with a hardware missing operator if it is missing; wherein, the hardware missing operator refers to the CPU operator in the target operator that is not configured in the current hardware. The module 730 is used to call the hardware missing operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system if the inference engine interfaced with the current hardware is not configured with the hardware missing operator. Registration module 740 is used to register all the target operators to the inference engine and / or the current hardware.

[0075] In an optional embodiment of this application, the second judgment module 720 is further configured to: if the inference engine interfaced with the current hardware is configured with the hardware missing operator, determine whether the computational data type required by the model to be computed is in a preset set of CPU-supported computational data types; if the computational data type required by the model to be computed is in the preset set of CPU-supported computational data types, then based on the data of the computational data type, perform inference computation on the model to be computed by calling the hardware missing operator configured in the inference engine and other target operators in the current hardware through the hardware extension plug-in system.

[0076] In an optional embodiment of this application, the second judgment module 720 is further configured to: if the data type of the computation required by the model to be computed is not in the preset set of CPU-supported computation data types, modify the data of the computation data type to CPU-compatible computation data of any data type in the preset set of CPU-supported computation data types; and based on the CPU-compatible computation data, perform inference computation on the model to be computed by calling the hardware missing operator configured in the inference engine and other target operators in the current hardware through the hardware extension plug-in system.

[0077] In an optional embodiment of this application, the calling module 730 is further configured to: if the inference engine interfaced with the current hardware is not configured with the hardware missing operator, determine whether the computational data type required by the model to be computed is in a preset set of CPU-supported computational data types; if the computational data type required by the model to be computed is in the preset set of CPU-supported computational data types, then, based on the data of the computational data type, call the hardware missing operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0078] In an optional embodiment of this application, the calling module 730 is further configured to: if the data type of the computation required by the model to be computed is not in the preset set of CPU-supported computation data types, modify the data of the computation data type to CPU-compatible computation data of any data type in the preset set of CPU-supported computation data types; and based on the CPU-compatible computation data, call the missing hardware operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

[0079] In an optional embodiment of this application, the plug-in definition module includes: a plug-in interface layer providing standardized API interfaces, a hardware abstraction layer, and a hardware driver interface layer. The calling module 730 is further configured to: obtain configuration information required by the inference engine for inference computation through the plug-in interface layer; the configuration information includes at least: the target operator required by the model to be computed and the data type of the computation; execute the interface functions corresponding to each of the standardized API interfaces through the hardware abstraction layer; the interface functions refer to the task functions executed by the inference engine when performing inference computation; encapsulate the usage content of peripherals through the hardware driver interface layer and provide the CPU with an interface to call the standard display driver model.

[0080] In one optional embodiment of this application, the standardized API interface includes at least one of the following: an interface for registering hardware-supported operators, an interface for allocating hardware memory, a host-to-device data transfer interface, an interface for executing hardware operators, a weight pre-download interface, a missing operator list interface, and a weight mode setting interface.

[0081] In one optional embodiment of this application, the hardware abstraction layer includes at least one of the following: an operator mapper, a memory manager, a data synchronizer, and a rollback checker.

[0082] In an optional embodiment of this application, the registration module 740 is further configured to: obtain a weight list of the current weight mode of the model to be calculated through a weight mode setting interface; obtain and register all target operators through a hardware-supported operator registration interface, and receive a hardware-supported operator list and a hardware-unsupported operator list fed back by the CPU; if the target operator is in the hardware-supported operator list, pre-download the weights in the weight list to the hardware driver interface layer in the plug-in definition module through a weight pre-download interface; allocate memory for the current device through a hardware memory allocation interface, and store the weight list.

[0083] In an optional embodiment of this application, the registration module 740 is further configured to obtain a list of missing operators through a missing operator list interface; wherein the list of missing operators contains all the hardware missing operators; and call the CPU rollback operator module to roll back to the CPU for inference calculation by registering all target operators using CPU operators.

[0084] In an optional embodiment of this application, the hardware extension plugin module further includes: a model weight module; the running device 700 of the open-source engine further includes a testing module, which is used to perform end-to-end performance testing on the model to be computed; the testing process includes: The model to be computed is subjected to single-layer transformer weight processing; performance testing is performed on the model based on the single-layer transformer weight processing. The model to be computed is then subjected to two-layer transformer weight processing and / or expert number partitioning; parallelism testing is performed on the model based on the two-layer transformer weight processing and / or expert number partitioning.

[0085] Specific limitations regarding the aforementioned open-source engine runtime device 700 can be found in the above description of the limitations on the open-source engine's execution method, and will not be repeated here. Each module in the aforementioned open-source engine runtime device 700 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0086] In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements one of the above-described open-source engine operation methods. This includes: memory and a processor; the memory stores the computer program; and the processor executes the computer program to implement any step of the above-described open-source engine operation method.

[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can perform any step of the above-described method of running the open-source engine.

[0088] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0093] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for running an open-source engine, characterized in that, A hardware extension plugin system applied to an open-source engine, the hardware extension system comprising at least: an inference engine module and a hardware extension plugin module; the inference engine module is configured with a model to be computed and an inference engine; the hardware extension plugin module comprising at least: a plugin definition module and a CPU rollback operator module; the method comprising: Determine whether the current hardware contains all the target operators required for inference calculations of the model to be computed. If missing, it is determined whether the inference engine interfaced with the current hardware is configured with a hardware missing operator; wherein, the hardware missing operator refers to the CPU operator in the target operator that is not configured in the current hardware; If the inference engine that interfaces with the current hardware is not configured with the missing hardware operator, the missing hardware operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system. Register all the target operators to the inference engine and / or the current hardware.

2. The method for running the open-source engine according to claim 1, characterized in that, After determining whether the inference engine interfaced with the current hardware is configured with a hardware missing operator, the method further includes: If the inference engine that interfaces with the current hardware is configured with the hardware missing operator, then determine whether the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU. If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator configured in the inference engine and other target operators in the current hardware are called through the hardware extension plug-in system to perform inference computation on the model to be computed.

3. The method for running the open-source engine according to claim 2, characterized in that, After determining whether the inference engine interfaced with the current hardware is configured with a hardware missing operator, the method further includes: If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU-compatible algorithm, the hardware missing operator configured in the inference engine and other target operators in the current hardware are invoked through the hardware extension plugin system to perform inference calculations on the model to be calculated.

4. The method for running the open-source engine according to claim 1, characterized in that, If the inference engine interfaced with the current hardware is not configured with a hardware missing operator, then the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system, including: If the inference engine that interfaces with the current hardware is not configured with the missing hardware operator, then determine whether the data type of the computation required by the model to be computed is in the preset set of data types of computation supported by the CPU. If the data type of computation required by the model to be computed is in the preset set of data types of computation supported by the CPU, then based on the data of the data type of computation, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

5. The method for running the open-source engine according to claim 4, characterized in that, If the inference engine interfaced with the current hardware is not configured with a hardware missing operator, then the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system, which further includes: If the data type of computation required by the model to be computed is not in the preset set of CPU-supported data types, then the data of the data type of computation is modified to CPU-compatible data of any data type in the preset set of CPU-supported data types. Based on the data calculated using the CPU compatibility, the hardware missing operator is called from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system.

6. The method for running the open-source engine according to claim 1, characterized in that, The plugin definition module includes: a plugin interface layer providing standardized API interfaces, a hardware abstraction layer, and a hardware driver interface layer; wherein, the method further includes: The configuration information required for the inference engine to perform inference calculations is obtained through the plugin interface layer; the configuration information includes at least: the target operator required by the model to be calculated and the data type of the calculation. The hardware abstraction layer executes the interface functions corresponding to each of the standardized API interfaces; the interface functions refer to the task functions executed by the inference engine when performing inference calculations. The hardware driver interface layer encapsulates the usage of peripherals and provides the CPU with an interface to call the standard display driver model.

7. The method for running the open-source engine according to claim 6, characterized in that, The standardized API interface includes at least one of the following: an interface for registering hardware-supported operators, an interface for allocating hardware memory, an interface for host-to-device data transfer, an interface for executing hardware operators, an interface for pre-downloading weights, an interface for listing missing operators, and an interface for setting weight modes.

8. The method for running the open-source engine according to claim 7, characterized in that, The hardware abstraction layer includes at least one of the following: an operator mapper, a memory manager, a data synchronizer, and a rollback checker.

9. The method for running the open-source engine according to claim 7, characterized in that, Registering all the target operators to the inference engine and / or the current hardware includes: The weight list of the current weight mode of the model to be calculated can be obtained through the weight mode setting interface; Obtain and register all target operators by registering the hardware-supported operator interface, and receive the list of hardware-supported operators and the list of hardware-unsupported operators fed back by the CPU. If the target operator is in the list of operators supported by the hardware, the weights in the weight list are pre-downloaded to the hardware driver interface layer in the plug-in definition module through the weight pre-download interface. The memory of the current device is allocated through the hardware memory allocation interface, and the weight list is stored.

10. The method for running the open-source engine according to claim 9, characterized in that, The step of registering all the target operators to the inference engine and / or the current hardware further includes: The missing operator list is obtained through the missing operator list interface; wherein, the missing operator list contains all the hardware missing operators; The CPU rollback operator module is invoked to roll back to the CPU for inference calculation by registering all target operators with CPU operators; wherein, the list of all target operators includes the list of missing operators.

11. The method for running the open-source engine according to claim 1, characterized in that, The hardware extension plugin module further includes: a model weight module; the method further includes end-to-end performance testing of the model to be calculated; the testing process includes: The model to be computed is subjected to single-layer transformer weight processing; The performance of the model to be computed is tested based on the weight processing of the single-layer transformer.

12. The method for running the open-source engine according to claim 1, characterized in that, The hardware extension plugin module further includes: a model weight module; the method further includes end-to-end performance testing of the model to be calculated; the testing process includes: The model to be computed is subjected to weight processing using a two-layer transformer, and / or expert number segmentation processing. Based on the weight processing of the two-layer transformer and / or the expert number segmentation processing, the parallelism of the model to be computed is tested.

13. A device for running an open-source engine, characterized in that, A hardware extension plugin system for an open-source engine, the hardware extension system comprising at least: an inference engine module and a hardware extension plugin module; the inference engine module is configured with a model to be computed and an inference engine; the hardware extension plugin module comprising at least: a plugin definition module and a CPU rollback operator module; the device comprising: The first judgment module is used to determine whether the current hardware is configured with all the target operators required for the inference calculation of the model to be calculated. The second judgment module is used to determine whether the inference engine interfaced with the current hardware is configured with a hardware missing operator if the missing operator is missing; wherein, the hardware missing operator refers to the CPU operator among the target operators that are not configured in the current hardware. The calling module is used to call the hardware missing operator from the CPU operator library pre-configured in the CPU through the hardware extension plug-in system if the inference engine interfaced with the current hardware is not configured with the hardware missing operator. A registration module is used to register all the target operators to the inference engine and / or the current hardware.

14. A computer device, comprising: A memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images