A method for managing inference engines that automatically handle model and runtime dependencies.

Abstract

This invention discloses a method for managing inference engines that automatically handles models and runtime dependencies, relating to the field of artificial intelligence inference engine management technology. By constructing a top-down three-layer system architecture and defining standardized interfaces and factory patterns, this invention achieves unified management of the entire model lifecycle and automatic resolution and installation of runtime dependencies. Simultaneously, this method supports plug-in extensions of heterogeneous inference engines, allowing new inference engines to be incorporated without modifying the upper-layer business scheduling layer code, significantly improving system flexibility and maintainability. Furthermore, through a preset dynamic mapping formula for performance parameters, this method can dynamically adjust the performance parameters of the inference engine according to different hardware environments, achieving optimal performance and improving system adaptability and resource utilization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference engine management technology, specifically a method for managing inference engines that automatically process model and runtime dependencies. Background Technology

[0002] In the field of artificial intelligence, with the rapid development and widespread application of deep learning technology, various complex models have been developed to address diverse application scenarios, such as natural language processing, image recognition, and speech recognition. The deployment and operation of these models often depend on specific inference engines and runtime environments, and there may be complex dependencies between different models and inference engines.

[0003] In traditional technologies, the management of models and inference engines typically faces several major challenges: First, the resolution and installation of model and runtime dependencies are cumbersome and require manual intervention, which not only increases deployment time but also raises the error rate. Second, the dependencies between different models and inference engines are complex and variable, making unified management and scheduling difficult, resulting in poor system flexibility and maintainability. Third, adding a new inference engine often requires modifying the code of the upper-level business scheduling layer to adapt to the new engine, which not only increases development costs but also limits the system's scalability. Finally, traditional methods struggle to dynamically adjust the performance parameters of the inference engine according to different hardware environments, leading to low resource utilization and an inability to fully leverage the hardware's potential.

[0004] In view of the shortcomings of traditional technologies in model and runtime dependency management, the present invention proposes an inference engine management method that automatically handles model and runtime dependencies, which is of particular importance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for managing inference engines that automatically handles model and runtime dependencies. This method achieves unified management of the entire model lifecycle and automatic resolution and installation of runtime dependencies by constructing a top-down three-layer system architecture and defining standardized interfaces and factory patterns. Simultaneously, this method supports plug-in extensions of heterogeneous inference engines, allowing new inference engines to be incorporated without modifying the upper-layer business scheduling code, significantly improving system flexibility and maintainability. Furthermore, through a preset dynamic mapping formula for performance parameters, this method can dynamically adjust the performance parameters of the inference engine according to different hardware environments to achieve optimal performance, improving system adaptability and resource utilization.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a method for managing inference engines that automatically handle model and runtime dependencies, the specific steps of which are as follows: S1. System Architecture Initialization and Interface Specification Definition: Construct a top-down three-layer system architecture, namely the business scheduling layer, the runtime abstraction layer, and the engine implementation layer; define standardized runtime interfaces RuntimeInterface and RuntimeFactoryInterface in the runtime abstraction layer, and initialize the runtime factory manager designed with the singleton pattern. S2. Runtime Factory Registration and Instantiation Management: When the system starts, the runtime factory manager performs initialization, obtains the full model list and runtime list through the manifest loader, creates corresponding runtime factory instances for each heterogeneous inference engine type and completes registration; when the business scheduling layer initiates a call request, it generates a runtime instance that conforms to the runtime interface specification through the instance creation method of the corresponding runtime factory. S3. Automatic Model Installation and Runtime Dependency Handling: After receiving the model installation request, the business scheduling layer executes a phased model installation process. The model installation process includes an independent runtime assurance phase, in which the runtime dependency information corresponding to the target model is automatically parsed, and the installation status and version matching degree of the corresponding runtime are verified. If the verification determines that the corresponding runtime is not installed or the version is mismatched, the fully automated installation of the runtime is automatically triggered, and after completion, the model installation execution phase begins. S4. Unified scheduling of the entire model lifecycle: The business scheduling layer initiates calls to the runtime instance of the underlying engine implementation layer through the standardized interface of the runtime abstraction layer to complete the full lifecycle management of model startup, shutdown, status monitoring, and uninstallation, and provides a unified inference service entry point to the outside world.

[0007] Furthermore, the runtime interface defined in S1 encapsulates the full capabilities of the inference engine, and sets up a total of 7 core method groups, namely the basic information acquisition method group, lifecycle management method group, operation control method group, health check and migration method group, configuration management method group, status query method group, and model management Hook method group. The basic information acquisition method group is used to obtain the full basic information of the runtime and the corresponding model. The lifecycle management method group is used for standardized management and control of the entire process of runtime installation and uninstallation. The operation control method group is used for the start and stop operations of runtime services and corresponding models. The health check and migration method group is used for runtime status verification and cross-device migration of storage directories. The configuration management method group is used for unified management and persistent storage of runtime configuration. The status query method group is used to obtain the runtime's occupancy status and environment adaptation status. The model management Hook method group is used to insert custom engine logic at key nodes in the entire process of model management.

[0008] Furthermore, this method establishes a unified configuration management metadata framework, defining standardized configuration structures and configuration metadata specifications for all runtime environments. The configuration metadata defines the description information, data type, default value, optional value range, and validation rules for each configuration item. Based on the configuration metadata, the configuration management framework implements three core capabilities: configuration item validity validation, persistent storage, and hot reloading. Validity validation is automatically performed when a configuration item is updated, and configuration items that do not meet the validation rules are directly intercepted and the corresponding prompt is returned. Persistent storage writes configuration items uniformly to a local standardized configuration file, supporting the import and export of configuration files. The hot reloading capability completes the update and activation of configuration items without restarting the runtime service. At the same time, the configuration metadata provides a unified configuration display specification for the front-end interface, realizing standardized interface rendering for different runtime configuration items.

[0009] Furthermore, the runtime factory manager initialized in S1 adopts a singleton pattern design, generating only one unique instance globally. Internally, the runtime factory manager maintains a mapping table from runtime types to factory instances. The mapping table stores data using a key-value pair structure, where the key is a unique identifier for the runtime type and the value is the instantiated object of the corresponding runtime factory. The runtime factory manager provides three core standardized capabilities: factory registration, factory query, and factory deregistration. The factory registration capability is used to include newly added runtime factory instances in global management, the factory query capability is used to quickly obtain the corresponding factory instance based on the runtime type, and the factory deregistration capability is used to remove invalid runtime factories from the global mapping table. When the system starts, the runtime factory manager automatically executes the default factory registration process, completing the instantiation and registration operations of all factories corresponding to the built-in inference engines.

[0010] Furthermore, the fully automated installation process in S3 adopts a phased state machine design, consisting of three core phases: preparation, execution, and completion. The preparation phase sequentially performs system environment checks, disk space verification, and system dependency detection. If a missing dependency is detected, the corresponding dependency is automatically installed and configured. After all verifications pass, the process proceeds to the next phase. The execution phase sequentially performs four operations: runtime installation package download, SHA256 hash value verification, installation package decompression, and engine-specific installation logic execution. The download process supports breakpoint resumption, and a re-download process is automatically triggered if verification fails. The completion phase sequentially performs three operations: configuration file writing, symbolic link creation, and runtime health check. After the health check passes, the runtime status is updated to "installed." Each phase supports independent interruption and recovery, and each key node has a corresponding Hook interface.

[0011] Furthermore, the model installation process in S3 is divided into four independent stages: preparation, runtime assurance, execution, and completion. The preparation stage receives and parses the model installation request initiated by the user, and obtains the basic information, runtime dependency information, and hardware adaptation requirements of the target model from the global model list. The runtime assurance stage automatically parses the runtime type and version requirements corresponding to the target model, verifies the installation status and version matching degree of the corresponding runtime, and automatically completes the installation and compatibility verification of missing runtimes. The execution stage calls the model installation method of the corresponding runtime instance, completes the download, hash verification, decompression, and directory configuration operations of the model file, provides real-time feedback on the installation progress and download speed through callback functions, completes the specific model configuration logic of the execution engine, updates the global model list and the local model database, and completes the entire model installation process.

[0012] Furthermore, the method also includes a hardware-aware runtime availability quantification step, whereby each runtime instance calculates its adaptation score in the current environment using a preset availability comprehensive scoring formula, and determines whether the runtime should participate in system scheduling based on the score. The availability comprehensive scoring formula is as follows: ,in The runtime is given a comprehensive availability score under the current hardware environment, with a score range of 0-100. A score of ≥60 indicates that the runtime is eligible for scheduling. The GPU is assigned a weight of 0.45. The CPU is assigned a weight of 0.2. The memory adaptation weight is set to 0.15. The driver version adaptation weight is set to 0.2. All weight coefficients were determined by multiple linear regression analysis of the engine running success rate data under 1200 different hardware environments. The GPU compatibility normalized value ranges from 0 to 100 and is calculated by weighting three indicators: GPU manufacturer compatibility, memory capacity satisfaction, and computing power compatibility. The CPU adaptability normalization value ranges from 0 to 100 and is calculated by normalizing the three indicators of CPU core count, thread count, and clock speed. This is a normalized value for system memory adaptability, ranging from 0 to 100, obtained by normalizing the ratio of available memory capacity to the minimum memory requirement at runtime. This is a normalized value for driver version compatibility, ranging from 0 to 100, calculated from the matching degree between the current driver version and the lowest compatible driver version at runtime.

[0013] Furthermore, the complete model startup process in S4 is as follows: After receiving the model startup request initiated by the user, the business scheduling layer first obtains the complete information of the target model from the local model database, parses it to obtain the runtime type corresponding to the model, then obtains the runtime factory of the corresponding type through the runtime factory manager, calls the instance creation method of the factory to generate a runtime instance that conforms to the interface specification, calls the information acquisition method of the runtime instance to verify the installation status of the runtime, directly returns the corresponding error information in the non-installed state, verifies the running status of the runtime in the installed state, calls the startup method to start the runtime service in the non-running state, and directly executes subsequent operations in the running state, calls the model startup method of the runtime instance to complete the entire process of model file verification, file location, parameter mapping, process startup, and health check, and finally provides a unified inference service interface to the outside world.

[0014] Furthermore, in the model startup process, the standardized performance options configured by the user are mapped to the native startup parameters of the corresponding inference engine through a preset dynamic performance parameter mapping formula. The dynamic performance parameter mapping formula is as follows: ,in These are the original startup parameter values ​​for the inference engine generated after mapping; This is the default baseline value for the parameter in the corresponding engine, which is obtained from the predefined value in the manifest file of the corresponding runtime. For parameter adjustment coefficients, fixed values ​​are set for different types of inference engines: 0.8 for the large language model inference engine, 0.6 for the speech recognition inference engine, and 0.9 for the image generation inference engine. Standardized performance level values ​​are configured for users, ranging from 1 to 5, corresponding to five levels: low power consumption, balanced, performance, high performance, and extreme performance. This is the minimum performance level, and its value is fixed at 1. This represents the available hardware resources of the current device. For CPU inference scenarios, it is the number of available CPU threads, and for GPU inference scenarios, it is the available GPU memory capacity, in GB. This represents the total hardware resources of the current device. For CPU inference scenarios, it represents the total number of CPU threads, and for GPU inference scenarios, it represents the total GPU memory capacity, in GB.

[0015] Furthermore, this method also supports pluggable extensions of heterogeneous inference engines. When adding a new inference engine, only three standardized operations need to be completed to be included in the unified management of the system. The first operation is for the implementation class of the new engine to complete the implementation of all standardized methods in the runtime interface and runtime factory interface, and encapsulate the unique business logic of the corresponding engine. The second operation is to compile the runtime manifest and model manifest corresponding to the new engine, defining the engine's basic information, version requirements, hardware adaptation rules, configuration item metadata, and supported model types in the manifest. The third operation is to register the runtime factory instance corresponding to the new engine to the global runtime factory manager when the system starts. After registration, the system can automatically recognize the new inference engine without modifying any code in the upper-level business scheduling layer, realizing hot-swappable extension of the engine.

[0016] Compared with existing technologies, this model and runtime-dependent automatic processing inference engine management method has the following advantages: I. This method constructs a top-down three-tier system architecture and defines standardized interfaces and factory patterns in the runtime abstraction layer, achieving unified management of the entire lifecycle of the model, including model installation, runtime dependency resolution and automatic installation, model startup, shutdown, status monitoring, and uninstallation. This automated management approach significantly improves the efficiency of model deployment and operation, reduces manual intervention, lowers the error rate, enables models to quickly respond to business needs, and enhances the overall system's flexibility and maintainability.

[0017] Second, this method defines standardized runtime interfaces and runtime factory interfaces, enabling the addition of new inference engines to be integrated into the system's unified management by only three standardized operations: interface implementation, manifest compilation, and factory registration. No modification to the upper-layer business scheduling code is required, achieving hot-swappable engine expansion. Simultaneously, through preset dynamic performance parameter mapping formulas, this method can dynamically generate the corresponding native startup parameters for the inference engine based on the user-configured standardized performance options and the current hardware resource status of the device. This achieves optimal performance under different hardware environments, improving system adaptability and resource utilization.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 A flowchart illustrating a method for managing an inference engine that automatically handles model and runtime dependencies; Figure 2 A phased flowchart of model installation and runtime dependency automatic processing for an inference engine management method that automatically processes model and runtime dependencies. Figure 3 This is a flowchart illustrating the model startup and dynamic performance parameter adaptation process for a model and runtime-dependent inference engine management method. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] Example This embodiment is applied to an intelligent security edge computing node in a park. The node is equipped with an x86 architecture industrial-grade CPU and an NVIDIA Jetson edge GPU, and connects to 16 high-definition video acquisition devices. It needs to run three types of AI inference tasks simultaneously and stably: a face detection model, a personnel boundary crossing behavior analysis model, and a smoke and fire recognition model. The three types of models are adapted to three heterogeneous inference engines: TensorRT 8.6, ONNX Runtime 1.16, and OpenVINO 2023.1, respectively. The core solution is to automatically adapt to the runtime dependencies of multiple engines at the edge, manage the entire lifecycle of the model, and optimize scheduling in scenarios with limited hardware resources.

[0023] The first step is system architecture initialization and interface specification definition. This step constructs a top-down three-layer system architecture: a business scheduling layer, a runtime abstraction layer, and an engine implementation layer. The business scheduling layer interfaces with the security platform's video stream scheduling and event alarm services. The runtime abstraction layer serves as the core adaptation layer, and the engine implementation layer encapsulates the underlying capabilities of various heterogeneous inference engines. In the runtime abstraction layer, standardized runtime interfaces `RuntimeInterface` and `RuntimeFactoryInterface` are defined. Simultaneously, a runtime factory manager designed using the singleton pattern is initialized. The defined runtime interfaces encapsulate the full capabilities of the inference engine, setting up seven core method groups: basic information retrieval methods, lifecycle methods, etc. The system comprises four method groups: Lifecycle Management, Operation Control, Health Check and Migration, Configuration Management, Status Query, and Model Management Hook. The Basic Information Acquisition method group is used to obtain full basic information about the runtime and corresponding models. The Lifecycle Management method group is used for standardized control of the entire runtime installation and uninstallation process. The Operation Control method group is used for starting and stopping runtime services and corresponding models. The Health Check and Migration method group is used for runtime status verification and cross-device migration of storage directories. The Configuration Management method group is used for unified management and persistent storage of runtime configurations. The Status Query method group is used to obtain the runtime's occupancy status and environment adaptation status. The Model Management Hook method group is used to insert custom engine logic at key nodes throughout the entire model management process.

[0024] This step simultaneously sets up a unified configuration management metadata framework, defining a standardized configuration structure and configuration metadata specification for all runtime environments. The configuration metadata defines the description, data type, default value, optional value range, and validation rules for each configuration item. Based on this metadata, the configuration management framework implements three core capabilities: configuration item validity validation, persistent storage, and hot reloading. Validity validation is automatically performed when a configuration item is updated; configuration items that do not meet the validation rules are directly intercepted and a corresponding warning is returned. Persistent storage writes configuration items uniformly to a local standardized configuration file, supporting the import and export of configuration files. Hot reloading updates and applies configuration items without restarting the runtime service. The data provides a unified configuration and display standard for the front-end interface. The initialized runtime factory manager adopts a singleton pattern design, generating only one unique instance globally. Internally, the runtime factory manager maintains a mapping table from runtime types to factory instances. The mapping table stores data using a key-value pair structure, where the key is a unique identifier for the runtime type and the value is the instantiated object of the corresponding runtime factory. The runtime factory manager provides three core capabilities to the outside world: standardized factory registration, factory query, and factory deregistration. The factory registration capability is used to include newly added runtime factory instances in global management, the factory query capability is used to quickly obtain the corresponding factory instance based on the runtime type, and the factory deregistration capability is used to remove invalid runtime factories from the global mapping table.

[0025] The second step is runtime factory registration and instantiation management. When the system starts, the runtime factory manager performs initialization, obtains the full model list and runtime list through the manifest loader, creates corresponding runtime factory instances for the three heterogeneous inference engine types TensorRT, ONNXRuntime, and OpenVINO, and completes registration. The three types of runtime factory instances are included in the global mapping table management, and the instantiation and registration operations of the corresponding factories for all built-in inference engines are completed. During the instantiation process of the runtime factory, the verification of the engine's basic capabilities and the loading of adaptation rules are completed simultaneously. When the business scheduling layer initiates a call request for different security analysis models, the instance creation method of the corresponding runtime factory generates a runtime instance that conforms to the runtime interface specification. The instance generation process strictly follows the standardized specifications defined by the interface, shielding the implementation differences of different underlying inference engines.

[0026] The third step involves automatic model installation and runtime dependency handling. After receiving batch installation requests from users for face detection models, personnel boundary crossing analysis models, and fireworks recognition models, the business scheduling layer executes a phased model installation process. This process is divided into four independent phases: preparation, runtime assurance, execution, and completion. In the preparation phase, the system receives and parses the user-initiated model installation requests, retrieves the basic information, runtime dependency information, and hardware compatibility requirements of the three target models from the global model list, and performs preliminary verification of the preconditions for model installation. The runtime assurance phase, as an independent core step, automatically parses the runtime dependency information corresponding to the target models, verifies the installation status and version compatibility of the corresponding runtimes, and if the verification determines that the corresponding runtime is not installed or the version is incompatible, it automatically triggers the fully automated installation of that runtime. After completion, the system then enters the execution phase of model installation.

[0027] The fully automated installation process at runtime employs a phased state machine design, comprising three core phases: preparation, execution, and completion. The preparation phase sequentially performs system environment checks, disk space verification, and system dependency detection. If a missing dependency is detected, it automatically installs and configures the corresponding dependency. After all verifications pass, the process proceeds to the next phase. The execution phase sequentially performs four operations: runtime installation package download, SHA256 hash value verification, installation package decompression, and engine-specific installation logic execution. The download process supports breakpoint resumption, and a re-download is automatically triggered if verification fails. The completion phase sequentially performs three operations: configuration file writing, symbolic link creation, and runtime health check. After the health check passes, the runtime status is updated to "installed." Each phase supports independent interruption and resumption, and each key node has a corresponding Hook interface. The execution phase calls the model installation method of the corresponding runtime instance to complete the download, hash verification, decompression, and directory configuration of model files, providing real-time feedback on installation progress and download speed through callback functions. The completion phase executes engine-specific model configuration logic, updates the global model list and local model database, and completes the entire model installation process.

[0028] The fourth step is hardware-aware runtime availability quantification. Each runtime instance calculates its adaptation score in the current edge node hardware environment using a preset availability comprehensive scoring formula. Based on the score, it is determined whether the runtime should participate in system scheduling. The availability comprehensive scoring formula is as follows: ,in A comprehensive usability score for runtime under the current hardware environment; Adapt weights to the GPU. To adapt weights for the CPU, Weights are adapted for memory use. Adjust weights for driver versions; This is a normalized value for GPU adaptation. This is the CPU adaptability normalized value; This is the normalized value for system memory adaptability. To normalize the driver version compatibility value, the actual running status and compatibility of the edge node's GPU, CPU, system memory, and driver version are combined to complete the quantitative evaluation of the compatibility of each runtime. The runtime instance with the best compatibility is selected to participate in subsequent business scheduling, while runtime instances with unsatisfactory compatibility scores are blocked to avoid the ineffective occupation of edge node hardware resources.

[0029] The fifth step involves unified scheduling throughout the model's entire lifecycle. The business scheduling layer initiates calls to the runtime instances of the underlying engine implementation layer through the standardized interface of the runtime abstraction layer. This completes the full lifecycle management of the model, including startup, shutdown, status monitoring, and uninstallation. It also provides a unified entry point for inference services, connecting to video stream inference requests from the security business platform. The complete model startup process is as follows: After receiving the model startup request from the user, the business scheduling layer first obtains the complete information of the target model from the local model database, parses it to obtain the runtime type corresponding to the model, then obtains the runtime factory of the corresponding type through the runtime factory manager, calls the instance creation method of the factory to generate a runtime instance that conforms to the interface specification, calls the runtime instance information retrieval method to verify the installation status of the runtime. If the runtime is not installed, the corresponding error information is returned directly. If the runtime is installed, the running status is verified. If the runtime is not running, the startup method is called to start the runtime service. If the runtime is running, subsequent operations are executed directly.

[0030] The runtime instance's model startup method is invoked to complete the entire process of model file verification, file location, parameter mapping, process startup, and health check. During the model startup process, a preset dynamic performance parameter mapping formula maps user-configured standardized performance options to the corresponding inference engine's native startup parameters. This dynamic performance parameter mapping formula is as follows: ,in These are the original startup parameter values ​​for the inference engine generated after mapping; This is the default baseline value for this parameter in the corresponding engine; This refers to the parameter adjustment coefficient; Standardized performance level values ​​configured for users; This is the minimum performance level. This represents the available hardware resources of the current device. This involves considering the total hardware resources of the current device, adapting to the hardware resource usage and business performance requirements of edge nodes, and ultimately providing a unified inference service interface. This embodiment also supports plug-in extension of heterogeneous inference engines. If intelligent analysis models adapted to other inference engines are added later, only three standardized operations need to be completed to be included in the unified management of the system. The first operation is for the implementation class of the new engine to complete the implementation of all standardized methods in the runtime interface and runtime factory interface, and encapsulate the unique business logic of the corresponding engine. The second operation is to compile the runtime list and model list corresponding to the new engine, defining the engine's basic information, version requirements, hardware adaptation rules, configuration item metadata, and supported model types in the list. The third operation is to register the runtime factory instance corresponding to the new engine to the global runtime factory manager when the system starts. After registration, the system can automatically recognize the new inference engine without modifying the core architecture of the system.

[0031] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for managing inference engines that automatically handle model and runtime dependencies, characterized in that, The specific steps of this method are as follows: S1. System Architecture Initialization and Interface Specification Definition: Construct a top-down three-layer system architecture, namely the business scheduling layer, the runtime abstraction layer, and the engine implementation layer; define standardized runtime interfaces RuntimeInterface and RuntimeFactoryInterface in the runtime abstraction layer, and initialize the runtime factory manager designed with the singleton pattern. S2. Runtime Factory Registration and Instantiation Management: When the system starts, the runtime factory manager performs initialization, obtains the full model list and runtime list through the manifest loader, creates corresponding runtime factory instances for each heterogeneous inference engine type and completes registration; when the business scheduling layer initiates a call request, it generates a runtime instance that conforms to the runtime interface specification through the instance creation method of the corresponding runtime factory. S3. Automatic Model Installation and Runtime Dependency Handling: After receiving the model installation request, the business scheduling layer executes a phased model installation process. The model installation process includes an independent runtime assurance phase, in which the runtime dependency information corresponding to the target model is automatically parsed, and the installation status and version matching degree of the corresponding runtime are verified. If the verification determines that the corresponding runtime is not installed or the version is mismatched, the fully automated installation of the runtime is automatically triggered, and after completion, the model installation execution phase begins. S4. Unified scheduling of the entire model lifecycle: The business scheduling layer initiates calls to the runtime instance of the underlying engine implementation layer through the standardized interface of the runtime abstraction layer to complete the full lifecycle management of model startup, shutdown, status monitoring, and uninstallation, and provides a unified inference service entry point to the outside world.

2. The inference engine management method for automatic processing of model and runtime dependencies according to claim 1, characterized in that, The runtime interface defined in S1 encapsulates the full capabilities of the inference engine and abstracts them into seven core method groups: basic information acquisition method group, lifecycle management method group, operation control method group, health check and migration method group, configuration management method group, status query method group, and model management Hook method group. The basic information acquisition method group is used to obtain full basic information of the runtime and corresponding models. The lifecycle management method group is used for standardized management and control of the entire runtime installation and uninstallation process. The operation control method group is used for the start and stop operations of runtime services and corresponding models. The health check and migration method group is used for runtime status verification and cross-device migration of storage directories. The configuration management method group is used for unified management and persistent storage of runtime configurations. The status query method group is used to obtain the runtime's occupancy status and environment adaptation status. The model management Hook method group is used to insert custom engine logic at key nodes in the entire model management process.

3. The method for managing inference engines that automatically handle model and runtime dependencies according to claim 1, characterized in that, This method establishes a unified configuration management metadata framework, defining standardized configuration structures and configuration metadata specifications for all runtime environments. The configuration metadata defines the description, data type, default value, optional value range, and validation rules for each configuration item. Based on the configuration metadata, the configuration management framework implements three core capabilities: configuration item validity validation, persistent storage, and hot reloading. Validation is automatically performed when a configuration item is updated, and configuration items that do not meet the validation rules are directly intercepted and the corresponding prompt is returned. Persistent storage writes configuration items uniformly to a local standardized configuration file, supporting the import and export of configuration files. The hot reloading capability completes the update and application of configuration items without restarting the runtime service. At the same time, the configuration metadata provides a unified configuration display specification for the front-end interface.

4. The method for managing inference engines that automatically handle model and runtime dependencies according to claim 1, characterized in that, The runtime factory manager initialized in S1 adopts a singleton pattern design, generating only one unique instance globally. Internally, the runtime factory manager maintains a mapping table from runtime types to factory instances. The mapping table stores data in a key-value pair structure, where the key is a unique identifier for the runtime type and the value is the instantiated object of the corresponding runtime factory. The runtime factory manager provides three core standardized capabilities: factory registration, factory query, and factory deregistration. The factory registration capability is used to include newly added runtime factory instances in global management, the factory query capability is used to quickly obtain the corresponding factory instance based on the runtime type, and the factory deregistration capability is used to remove invalid runtime factories from the global mapping table. When the system starts, the runtime factory manager automatically executes the default factory registration process, completing the instantiation and registration operations of all factories corresponding to the built-in inference engines.

5. The method for managing inference engines that automatically handle model and runtime dependencies according to claim 1, characterized in that, The fully automated installation process in S3 adopts a phased state machine design, consisting of three core phases: preparation, execution, and completion. The preparation phase sequentially performs system environment checks, disk space verification, and system dependency detection. If a missing dependency is detected, the corresponding dependency is automatically installed and configured. After all verifications pass, the process proceeds to the next phase. The execution phase sequentially performs four operations: runtime installation package download, SHA256 hash value verification, installation package decompression, and engine-specific installation logic execution. The download process supports breakpoint resumption, and a re-download process is automatically triggered if verification fails. The completion phase sequentially performs three operations: configuration file writing, symbolic link creation, and runtime health check. After the health check passes, the runtime status is updated to "installed." Each phase supports independent interruption and recovery, and each key node has a corresponding Hook interface.

6. The method for managing inference engines that automatically handle model and runtime dependencies according to claim 1, characterized in that, The model installation process in S3 is divided into four independent stages: preparation, runtime assurance, execution, and completion. The preparation stage receives and parses the user-initiated model installation request, retrieves the target model's basic information, runtime dependencies, and hardware compatibility requirements from the global model list. The runtime assurance stage automatically parses the runtime type and version requirements corresponding to the target model, verifies the installation status and version matching of the corresponding runtime, and automatically completes the installation and compatibility verification of missing runtimes. The execution stage calls the model installation method of the corresponding runtime instance, completes the download, hash verification, decompression, and directory configuration operations of the model file, provides real-time feedback on installation progress and download speed through callback functions, executes the specific model configuration logic of the execution engine, updates the global model list and local model database, and completes the entire model installation process.

7. The method for managing an inference engine that automatically handles model and runtime dependencies according to claim 1, characterized in that, The method further includes a hardware-aware runtime availability quantification step, whereby each runtime instance calculates its adaptation score in the current environment using a preset availability comprehensive scoring formula, and determines whether the runtime should participate in system scheduling based on the score. The availability comprehensive scoring formula is as follows: ,in A comprehensive usability score for runtime under the current hardware environment; Adapt weights to the GPU. To adapt weights for the CPU, Weights are adapted for memory use. Adjust weights for driver versions; This is a normalized value for GPU adaptation. This is the CPU adaptability normalized value; This is the normalized value for system memory adaptability. This is a normalized value for driver version compatibility.

8. The method for managing inference engines that automatically handle model and runtime dependencies according to claim 1, characterized in that, The complete model startup process in S4 is as follows: After receiving the model startup request initiated by the user, the business scheduling layer first obtains the complete information of the target model from the local model database, parses it to obtain the runtime type corresponding to the model, then obtains the runtime factory of the corresponding type through the runtime factory manager, calls the instance creation method of the factory to generate a runtime instance that conforms to the interface specification, calls the information acquisition method of the runtime instance to verify the installation status of the runtime, directly returns the corresponding error information in the non-installed state, verifies the running status of the runtime in the installed state, calls the startup method to start the runtime service in the non-running state, and directly executes subsequent operations in the running state, calls the model startup method of the runtime instance to complete the entire process of model file verification, file location, parameter mapping, process startup, and health check, and finally provides a unified inference service interface to the outside world.

9. The method for managing an inference engine that automatically handles model and runtime dependencies according to claim 1, characterized in that, In the model startup process, the standardized performance options configured by the user are mapped to the native startup parameters of the corresponding inference engine through a preset dynamic performance parameter mapping formula. The dynamic performance parameter mapping formula is as follows: ,in These are the original startup parameter values ​​for the inference engine generated after mapping; This is the default baseline value for this parameter in the corresponding engine; This refers to the parameter adjustment coefficient; Standardized performance level values ​​configured for users; This is the minimum performance level. This represents the available hardware resources of the current device. This represents the total hardware resource value of the current device.

10. The inference engine management method for automatic processing of model and runtime dependencies according to claim 1, characterized in that, This method also supports plug-in extension of heterogeneous inference engines. When adding a new inference engine, only three standardized operations need to be completed to be included in the unified management of the system. The first operation is for the implementation class of the new engine to complete the implementation of all standardized methods in the runtime interface and runtime factory interface, and encapsulate the unique business logic of the corresponding engine. The second operation is to compile the runtime manifest and model manifest corresponding to the new engine, and define the engine's basic information, version requirements, hardware adaptation rules, configuration item metadata, and supported model types in the manifest. The third operation is to register the runtime factory instance corresponding to the new engine to the global runtime factory manager when the system starts. After the registration is completed, the system can automatically recognize the new inference engine.

Images