Method, device and storage medium for controlling end-side cross-platform large model inference engine

Abstract

The application discloses an end-side cross-platform large model reasoning engine control method, device and storage medium, and relates to the technical field of model management. The method comprises the following steps: receiving a model loading request, obtaining resource requirement information of a target model and hardware resource available information of a terminal device; performing pre-checking on the target model based on the resource requirement information and the hardware resource available information; determining a target reasoning backend and a target running device for running the target model according to capability description information of a plurality of heterogeneous reasoning backends; loading the target model on the target running device based on a running mode of the target reasoning backend through a unified scheduling layer, and generating a general session identifier decoupled from a backend implementation; receiving a reasoning request through the general session identifier, scheduling the target reasoning backend to perform reasoning on the target running device, and returning a reasoning result. The application improves the execution efficiency of the end-side reasoning engine on different platforms.

Description

Technical Field

[0001] This application relates to the field of model management technology, and in particular to a control method, device and storage medium for a cross-platform large model inference engine on the edge. Background Technology

[0002] The model platforms in related technologies are mainly optimized for a single backend. For example, llama.cpp has good local inference capabilities, Ollama, LM Studio, and Jan.ai have relatively complete local model management and deployment experience on desktop, MLC LLM has strong support for mobile devices and multiple platforms, and vLLM is representative for high-throughput inference. The model platforms in these solutions usually have good adaptation depth and maturity on their respective bound backend frameworks, but they lack a fine-grained matching mechanism based on hardware resources before model loading. Users often need to rely on experience to judge whether the device can handle the model, and often reserve too many resources to avoid loading failures, resulting in resource over-provisioning and low device utilization. Summary of the Invention

[0003] The main purpose of this application is to provide a control method, device and storage medium for a cross-platform large model inference engine on the client side, which aims to solve the technical problem in related technologies that often reserve too many resources to avoid loading failures, resulting in resource over-provisioning.

[0004] To achieve the above objectives, this application provides a control method for a device-side cross-platform large model inference engine, the control method of which includes: Receive a model loading request, which is used to request the loading of the target model; Obtain the resource requirement information of the target model and the hardware resource availability information of the terminal device, perform domain-based budgeting on the hardware resource availability information, and determine the available resource quantity of each memory domain; Based on the resource requirement information and the available resources of each memory domain, the target model is pre-tested; Upon passing the pre-inspection, the target inference backend for running the target model and the target running device corresponding to the target inference backend are determined based on the capability description information of the multiple pre-registered heterogeneous inference backends. Through a unified scheduling layer, based on the operating mode of the target inference backend, the target model is loaded on the target running device, and a general session identifier decoupled from the backend implementation is generated. The inference request is received through the general session identifier, the target inference backend is scheduled to perform inference on the target running device and return the inference result.

[0005] In one embodiment, before receiving the model loading request, the method further includes: By using a cross-platform unified abstraction layer, multiple heterogeneous inference backends can be discovered by scanning specified paths or parsing configuration files on the operating system running on the terminal device. Obtain the registration information submitted by each of the heterogeneous inference backends, wherein the registration information includes at least the backend identifier, the supported operating system, the operating mode, and the supported computing device type; Based on the registration information of each heterogeneous inference backend, a structured backend registry is constructed as the capability description information of each of the multiple heterogeneous inference backends.

[0006] In one embodiment, the pre-detection of the target model based on the resource requirement information and the available resource quantity of each memory domain includes: Extract the model volume and quantization method of the target model from its metadata; Based on the model volume and the quantization method, the peak resource requirements of the target model in each memory domain are determined respectively. Based on the preset safety margin coefficient, the peak resource requirements of each memory domain are adjusted upwards to obtain the safety requirement value corresponding to each memory domain. The security requirement value corresponding to each memory domain is compared with the available resource quantity of the corresponding memory domain. If the security requirement value of any memory domain exceeds the available resources of the corresponding memory domain, the pre-check is determined to have failed, the model loading request is rejected, and a prompt message containing the resource gap between each memory domain is output. If the security requirements of all memory domains do not exceed the available resources of the corresponding memory domain, the pre-check is deemed successful, and resource reservation is performed for each memory domain.

[0007] In one embodiment, determining the target inference backend for running the target model and the target running device corresponding to the target inference backend based on the capability description information of each of the pre-registered heterogeneous inference backends includes: Based on the capability description information of each of the pre-registered heterogeneous inference backends, the operating system, model format, and computing device type of each heterogeneous inference backend are extracted; Based on the operating system of the terminal device and the model format of the target model, candidate inference backends are selected from the multiple heterogeneous inference backends; The performance scores of each candidate inference backend on different computing device types are obtained, and the weighted scores of each candidate inference backend are calculated in combination with the current available resource balance of the terminal device to obtain the backend sequence. The candidate inference backend with the highest weighted score is selected from the backend sequence as the target inference backend, and the computing device corresponding to the weighted score among the computing device types supported by it is determined as the target running device.

[0008] In one embodiment, the target inference backend's operating mode includes an embedded driver mode and an external process mode. The step of loading the target model on the target running device through a unified scheduling layer, based on the target inference backend's operating mode, and generating a universal session identifier decoupled from the backend implementation, includes: When the target inference backend is running in the embedded driver mode, the loading function of the target inference backend is called in the engine process to load the target model into the memory of the target running device and generate the general session identifier that is decoupled from the backend implementation. When the target inference backend is running in the external process mode, a child process corresponding to the target inference backend is created, an inter-process communication channel is established between the child process and the child process, a loading instruction and the path information of the target model are sent to the child process through the inter-process communication channel, and after receiving the ready confirmation returned by the child process, a general session identifier that is decoupled from the backend is generated.

[0009] In one embodiment, when the target inference backend is running in the external process mode, the step of creating a subprocess corresponding to the target inference backend includes: If the terminal device is running Windows, the child process is assigned to a job object, so that the lifecycle of the child process is bound to the job object; If the terminal device is running a Unix operating system, then the process identifier of the child process is recorded, and a mechanism is set for the child process to receive a termination signal when the engine process exits.

[0010] In one embodiment, after creating the subprocess corresponding to the target inference backend, the process includes: If the engine process exits abnormally, and the terminal device is running Windows, the operating system will destroy the job object corresponding to the engine process, and terminate all child processes assigned to the job object by destroying the job object. If the engine process exits abnormally and the terminal device is running a Unix operating system, the child process will detect the engine process's exit through a termination signal mechanism and terminate itself according to the termination signal mechanism.

[0011] In one embodiment, receiving an inference request via the universal session identifier, scheduling the target inference backend to perform inference on the target running device, and returning the inference result includes: Based on the general session identifier, the backend context corresponding to the general session identifier is found. The backend context includes at least the running mode of the target inference backend, and when the running mode is external process mode, the backend context also includes a communication channel. When the operating mode is the embedded driver mode, the inference interface of the target inference backend is called to obtain the inference result; When the running mode is external process mode, the inference request is serialized and sent to the subprocess corresponding to the target inference backend through the communication channel, and the inference result returned by the subprocess is received. The obtained inference results will be returned to the caller in a unified data format.

[0012] In addition, to achieve the above objectives, this application also provides a control device for a device-side cross-platform large model inference engine. The control device for the device-side cross-platform large model inference engine includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the control method for the device-side cross-platform large model inference engine as described above.

[0013] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a program for implementing a control method for a cross-platform large model inference engine is stored. The program for implementing the control method for a cross-platform large model inference engine is executed by a processor to implement the steps of the control method for a cross-platform large model inference engine as described above.

[0014] This application provides a control method for a cross-platform large model inference engine on the terminal side. First, by acquiring the available hardware resources of the terminal device and the resource requirements of the target model, a pre-check is performed on the target model. This pre-check compares the resource requirements with the available hardware resources to determine if the terminal device has the capability to run the target model. If the pre-check fails, it indicates that the terminal device does not have the capability to run the target model, and the model loading request is directly rejected with a prompt message indicating a resource gap, to avoid memory overflow or loading failure due to blind loading. If the pre-check passes, it indicates that the terminal device has the capability to run the target model. Then, based on the capability descriptions of multiple pre-registered heterogeneous inference backends, the target inference backend for running the target model and the corresponding target running device are determined, ensuring that the target model can be allocated to a device with the appropriate resources. Instead of being bound to a single backend framework as in related technologies, the target inference backend and runtime device are adapted to the source requirements and format characteristics, thus overcoming the performance loss caused by binding to a single backend in cross-platform operation. After determining the target inference backend and the target runtime device, the target model is loaded on the target runtime device through a unified scheduling layer based on the runtime mode of the target inference backend, and a general session identifier decoupled from the backend implementation is generated. This allows the upper-layer caller to initiate inference requests through the general session identifier without being aware of the specific implementation differences of the target inference backend, thereby shielding the call complexity caused by the heterogeneity of multiple backends. After generating the general session identifier, the inference request is received through the general session identifier, the target inference backend is scheduled to perform inference on the target runtime device and return the inference result, thus completing transparent inference across backends under a unified interface.

[0015] In summary, this application reduces the risk of memory overflow caused by blind loading by obtaining the resource requirements of the target model and the available hardware resources of the terminal device and performing domain-based pre-checks. When resources are insufficient, loading is rejected and a resource gap warning is output. When resources are sufficient, resources are reserved for each memory domain. After the pre-check is passed, a runtime plan is generated and the target inference backend and target running device are determined based on the capability description information of multiple heterogeneous inference backends. This allows the model to be allocated to an inference backend and running device that matches its resource requirements, reducing the degree of resource over-provisioning caused by binding to a single backend framework. A common session identifier is generated through a unified scheduling layer and decoupled from the backend implementation. The target inference backend is scheduled to perform inference using this common session identifier. A rollback operation is triggered when loading fails. When inference fails or the generated result is abnormal, invalidation processing is performed on the generated content, improving the operational stability of the edge inference engine under abnormal scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an embodiment of the control method for a cross-platform large model inference engine on the client side of this application; Figure 2 A flowchart is provided for Embodiment 4 of the control method for the cross-platform large model inference engine on the client side of this application. Figure 3 This is a schematic diagram of the hardware operating environment involved in the control device of the cross-platform large model inference engine on the client side of this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Currently, model platforms in related technologies are mainly optimized for a single backend. For example, llama.cpp has good local inference capabilities, Ollama, LM Studio, and Jan.ai offer a relatively complete local model management and deployment experience on desktops, MLC LLM is strong in mobile and multi-platform support, and vLLM is representative in high-throughput inference. These model platforms typically have good adaptation depth and maturity on their respective bound backend frameworks, but performance is compromised when running across platforms.

[0023] The main solution of this application is as follows: receiving a model loading request, the model loading request being used to request the loading of a target model; obtaining the resource requirement information of the target model and the hardware resource availability information of the terminal device; performing a pre-check on the target model based on the resource requirement information and the hardware resource availability information; upon passing the pre-check, determining the target inference backend for running the target model and the target running device corresponding to the target inference backend based on the capability description information of multiple pre-registered heterogeneous inference backends; loading the target model on the target running device through a unified scheduling layer based on the running mode of the target inference backend, and generating a general session identifier decoupled from the backend implementation; receiving inference requests through the general session identifier, scheduling the target inference backend to perform inference on the target running device and returning the inference result.

[0024] This application enables models to be allocated to inference backends and runtime devices that are compatible with their resource requirements, reducing the degree of resource overprovisioning caused by being tied to a single backend framework.

[0025] It should be noted that the execution entity in this embodiment can be the control system of a cross-platform, edge-side large-model inference engine, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a control device for a cross-platform, edge-side large-model inference engine capable of performing the above functions. This embodiment does not specifically limit the specific implementation. The following description uses the control system of a cross-platform, edge-side large-model inference engine as the execution entity to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, Embodiment 1 of this application proposes a control method for a cross-platform large model inference engine on the device side. Please refer to... Figure 1 The control method for the edge-side cross-platform large model inference engine includes steps S10 to S60: Step S10: Receive a model loading request, which is used to request the loading of the target model.

[0027] In this embodiment, a model loading request refers to a call instruction initiated by an upper-layer application or client to specify a large language model file to be loaded. This instruction carries at least the model file path or a unique model identifier. The target model is the specific large language model file pointed to by the request, and its format can be a pre-trained model weight file and its configuration file.

[0028] As an optional implementation, the inference engine exposes a unified application programming interface (API) that includes a model loading function. When a user selects a local model file and triggers the loading action via a graphical interface or command-line tool, the upper-layer program calls this loading function, passing the absolute path of the model file in the terminal device's file system as a request parameter. The inference engine performs a validity check on this path at the interface layer. If the check passes, it resolves the path to an internal reference pointing to a specific model file, thereby determining the target model.

[0029] As an alternative implementation, the inference engine runs as a local daemon service, receiving model loading requests by listening on a predefined local network port or message queue. The request message is encapsulated in a structured data format, containing the model identifier and version tag from the model repository. Upon receiving the message, the engine deserializes it to obtain the identifier and looks up the corresponding model storage path and metadata file location in the model repository's index table, thereby locating the target model entity. This approach decouples the initiation of model loading requests from the engine's main logic, enabling multiple clients to concurrently submit loading tasks without blocking the engine's main thread, thus improving system responsiveness.

[0030] Step S20: Obtain the resource requirement information of the target model and the hardware resource availability information of the terminal device, perform domain-based budgeting on the hardware resource availability information, and determine the available resource quantity of each memory domain.

[0031] In this embodiment, resource requirement information refers to the estimated value of hardware resource usage by the target model during inference, including at least the peak memory requirement or peak video memory requirement at runtime. Hardware resource availability information for the terminal device refers to the free physical memory capacity and free video memory capacity reported by the operating system at the current moment.

[0032] In this embodiment, resource requirement information refers to the estimated value of different hardware resource usage by the target model during inference. A memory domain refers to the logical division of different types of memory resources on a terminal device, including physical memory domains and video memory domains. The physical memory domain corresponds to the system memory resources that the central processing unit can schedule, while the video memory domain corresponds to the dedicated video memory resources that the graphics processing unit can schedule. Domain budgeting refers to breaking down the available hardware resource information according to the classification of memory domains, determining the upper limit of the currently allocable resources for each memory domain, and the available resource amount for each memory domain is the specific value obtained after domain budgeting.

[0033] As an optional implementation, the inference engine first reads the metadata segment from the header of the target model file, extracting structured fields such as model parameter size, weight tensor shape, and quantization precision level. Then, based on a pre-calibrated parameter-memory mapping table, the engine converts the extracted parameter size and quantization precision into peak runtime memory and peak GPU memory requirements. Simultaneously, the engine calls the system information query interface provided by the terminal device's operating system to obtain the current amount of free physical memory and free GPU memory. After obtaining this information, the engine uses the amount of free physical memory as the available resource in the physical memory domain and the amount of free GPU memory as the available resource in the GPU memory domain.

[0034] As an alternative implementation, a lightweight resource monitoring module resides within the inference engine. This module periodically collects the real-time memory and video memory status of the terminal device and maintains a valid snapshot of the most recently collected data. Upon receiving a model loading request, the engine reads the available physical memory and available video memory from this snapshot and performs a domain-based budgeting operation: allocating the available physical memory to the physical memory domain and the available video memory to the video memory domain, determining the upper limit of available resources for each domain. Regarding resource requirement information, the engine parses the model description file attached to the target model, extracting the declared peak physical memory and peak video memory requirements for runtime. If the model description file lacks these fields, a simulation graph compilation process is initiated. By traversing the model computation graph and accumulating the maximum temporary buffer size of each operator, the peak requirements for the physical memory domain and the peak requirements for the video memory domain are calculated respectively.

[0035] Step S30: Based on the resource requirement information and the available resource quantity of each memory domain, perform a pre-test on the target model.

[0036] Step S40: When the pre-inspection is passed, the target inference backend for running the target model and the target running device corresponding to the target inference backend are determined based on the capability description information of the multiple pre-registered heterogeneous inference backends.

[0037] In this embodiment, pre-detection refers to the logical judgment process of comparing the resource requirements of the target model with the currently available resources of the terminal device to determine whether the basic resource conditions for loading the model are met. Capability description information is a set of attributes of each inference backend registered during the engine startup phase, which includes at least the operating system type supported by the backend, the compatible model file format, and the type of computing device that can be scheduled. The target inference backend refers to the inference framework selected from the registered backends to actually execute the loading and inference of the target model, and the target running device refers to the physical computing unit, such as a central processing unit or graphics processing unit, that the backend will be bound to.

[0038] As an optional implementation, during pre-detection, the peak memory requirement in the resource demand information is directly compared with the available physical memory in the hardware resource availability information. If the available physical memory is not lower than the peak memory requirement, the pre-detection is considered successful; otherwise, the pre-detection is considered unsuccessful, and an error code is returned to the requester. After successful pre-detection, the engine selects a set of candidate backends from the capability description information table that meet two conditions: their supported operating system is consistent with the current terminal device's operating system, and their supported model format is compatible with the target model's file format. Next, the engine reads the preset performance priority of each candidate backend on each computing device, and combines this with the current idle level of each computing device, selecting the backend with the most abundant idle resources and the highest performance priority as the target inference backend, and simultaneously determining the computing device that the backend is preferentially bound to as the target running device. This method has low computational overhead and is suitable for scenarios with high real-time requirements.

[0039] As an alternative implementation, the pre-detection phase, in addition to comparing memory capacity, further assesses whether the memory bandwidth meets the requirements of model inference. The engine estimates the minimum memory bandwidth threshold based on the number of parameters in the target model and the typical inference batch size, and compares it with the memory bandwidth capability obtained by the terminal device through system queries. The pre-detection is considered successful only when both memory capacity and memory bandwidth meet the requirements. When determining the target inference backend, the engine introduces a model quantization method matching score for candidate backends: if a backend has native inference operators consistent with the target model's quantization method, it is given a higher matching score; if online dequantization is required at runtime, it is given a lower matching score. Finally, a weighted optimization is performed based on device load, performance priority, and quantization matching score to select the target inference backend and the target running device. This approach further avoids inference stuttering caused by insufficient memory bandwidth and prioritizes the use of native quantization support to reduce inference latency, improving the energy efficiency of edge inference.

[0040] Step S50: Through a unified scheduling layer, based on the operating mode of the target inference backend, the target model is loaded on the target running device, and a universal session identifier decoupled from the backend implementation is generated.

[0041] In this embodiment, the unified scheduling layer is an intermediate software layer located between the upper-layer application interface and the various underlying inference backends. Its responsibility is to adapt the loading and inference process according to the specific characteristics of the backend. The runtime mode describes how the target inference backend executes within the engine, including an embedded mode that executes within the engine process space via function library calls, and a separate mode that runs as an independent subprocess. The universal session identifier is a globally unique handle generated by the engine. Upper-layer applications use this handle to reference loaded model sessions without needing to know the underlying backend type and runtime mode.

[0042] As an optional implementation, the scheduling layer maintains a backend adaptation table, which records the running mode and loading method entry point for each backend identifier. Once the target inference backend is determined, the scheduling layer queries this adaptation table to obtain its running mode. If the running mode is embedded mode, the scheduling layer directly calls the model loading interface provided by the backend in the current thread of the engine, maps the target model file into the memory of the target running device, and obtains the model's internal context pointer returned by the backend. If the running mode is detached mode, the scheduling layer constructs a startup command line based on the executable file path of the target inference backend, creates a child process, establishes a full-duplex communication channel based on local sockets, and then sends the loading command and the target model path to the child process through the channel, waiting for the child process to return a confirmation signal that loading is ready. After successful loading, the scheduling layer generates an opaque universal session identifier, internally establishes a mapping relationship between this identifier and the actual backend context or communication channel, and returns this identifier to the upper-layer application.

[0043] As an alternative implementation, the scheduling layer is constructed using the session factory pattern. A unified session interface is predefined for different operating modes, and embedded session classes and separate session classes are implemented respectively. After determining the target inference backend, the factory method of the scheduling layer instantiates the corresponding session class object according to the operating mode. The embedded session class encapsulates direct calls to the backend's native API, while the separate session class encapsulates the inter-process communication protocol and data serialization logic. After completing model loading and initialization, the factory method maps the session class object handle to a generic integer session identifier. All subsequent operations performed using this identifier are dispatched to the specific implementation of the corresponding session class through virtual functions. This object-oriented design allows adding a new operating mode only requires adding a session subclass; the main logic of the scheduling layer remains unchanged, further enhancing extensibility.

[0044] Step S60: Receive an inference request through the general session identifier, schedule the target inference backend to perform inference on the target running device and return the inference result.

[0045] In this embodiment, an inference request refers to a computation request initiated by the upper-layer application after the model has been loaded, containing input data. The input data is typically a text sequence or tensor data that has undergone word segmentation. The inference result refers to the output data generated by the target model after performing forward computation on the input data, typically a generated text tag sequence or output tensor.

[0046] As an optional implementation, the upper-layer application invokes the engine's unified inference interface, passing in a generic session identifier and an input data buffer. The scheduling layer uses the generic session identifier as the key to look up its maintained session mapping table to obtain the corresponding session object. This session object internally records the backend type and operating mode. In embedded mode, the scheduling layer directly converts the input data into the tensor format expected by the backend, calls the backend's inference function, and blocks until the computation is complete to obtain the output tensor. In separate mode, the scheduling layer serializes the input data into a byte stream according to the agreed communication protocol and sends it to the child process through the communication channel held by the session object. After the child process completes inference, it serializes the result and sends it back. The scheduling layer receives this and deserializes it to obtain the output tensor. Regardless of the mode, the scheduling layer ultimately converts the output tensor into a unified data format agreed upon by the upper-layer application and returns the inference result.

[0047] As an alternative implementation, the inference request scheduling process employs an asynchronous, non-blocking design. Upon receiving an inference request, the engine immediately returns a future handle to the caller and simultaneously places the request in a priority queue. The scheduling layer's worker threads retrieve tasks from the queue based on request priority and a generic session identifier. For the split-mode backend, input data is passed to the child process via a shared memory region, avoiding the overhead of large-scale data serialization. After inference is complete, the output tensor is also returned via shared memory, written to the result buffer by the scheduling layer, and the upper-layer application is notified of inference completion via a callback function or event. This approach enables concurrent inference across multiple sessions, significantly improving engine throughput, and is particularly suitable for edge deployment scenarios that require simultaneous service of multiple interactive dialogues or batch requests.

[0048] For example, on the Apple Silicon chip platform, a user selects a 7B parameter large language model file using 4-bit quantization format on the application interface and clicks the load button. The application then calls the engine interface to initiate a model loading request, passing the path of the model file on the disk. After receiving the request, the engine parses the model file corresponding to the path and determines the target model to be loaded. The engine reads the parameter scale and quantization precision from the model file header and calculates the peak runtime memory requirement as 3.8GB based on the mapping table. Since the Apple Silicon platform uses a unified memory architecture, physical memory and video memory share the same address space. The engine learns through the system interface that the currently available unified memory is 6.2GB. After domain-based estimation, the engine uses this available unified memory amount as the available resource amount for both the physical memory domain and the video memory domain. After comparing the safety requirement value of each memory domain with the corresponding available resource amount, the engine determines that the pre-check is passed. The engine searches the list of registered inference backends and finds that the MLX backend supports the current macOS operating system and is compatible with this quantization model format, and is specifically optimized for the graphics processing unit of the Apple Silicon chip. The engine then identifies MLX as the target inference backend and the graphics processing unit as the target running device. The scheduling layer detects that the MLX backend is running in embedded mode and directly calls its loading interface to map model weights to a unified memory space. Leveraging the unified memory features of Apple Silicon, it avoids explicit data copying between the CPU and GPU, obtains the internal context pointer, and then generates a digital session identifier, returning it to the application. The user inputs text to be continued in the application, triggering the generation process. The application calls the inference interface, passing in the session identifier and the input text. The engine locates the internal context of the MLX backend through the session identifier, converts the input text into a tensor, calls the MLX inference function, performs forward computation on the GPU, obtains the output text tag sequence, decodes it into natural language text, and returns it to the application. The application interface displays the continuation result.

[0049] For example, on an NVIDIA chip platform, a user selects a 7B parameter large language model file using 4-bit quantization format on the application interface and clicks the load button. The application then calls the engine interface to initiate a model loading request, passing the path of the model file on the disk. After receiving the request, the engine parses the model file corresponding to the path to determine the target model to be loaded. The engine reads the parameter scale and quantization precision from the model file header, calculates the peak runtime memory requirement as 3.8GB and the peak VRAM requirement as 3.2GB based on the mapping table, and simultaneously learns through the system interface that the currently available physical memory is 10.5GB and the available VRAM is 6.8GB. The engine performs domain budgeting, using the 10.5GB of available physical memory as the available resource amount for the physical memory domain and the 6.8GB of available VRAM as the available resource amount for the VRAM domain. After adjusting the safety margin coefficient, the safe requirement value for the physical memory domain is 4.6GB, and the safe requirement value for the VRAM domain is 3.8GB. The engine compares the security requirements of each domain with the corresponding available resources. If both domains meet the requirements, the pre-check passes, and 4.6GB of resources are reserved for the physical memory domain and 3.8GB for the video memory domain. The engine searches the list of registered inference backends and finds that the llama.cpp backend supports the current Windows operating system and is compatible with the quantization model format, and supports accelerated inference via the CUDA backend using the NVIDIA graphics processor. The engine then identifies llama.cpp as the target inference backend and the NVIDIA graphics processor as the target running device. The scheduling layer finds that the llama.cpp backend is running in embedded mode, directly calls its loading interface to load the model weights from disk into physical memory for preprocessing, copies the tensors required for inference to video memory, obtains the internal context pointer, generates a digital session identifier, and returns it to the application. The user inputs code to be continued in the application and triggers completion. The application calls the inference interface, passing in the session identifier and the input text. The engine locates the internal context of the llama.cpp backend using the session identifier, converts the input text into a tensor, calls the CUDA inference function in llama.cpp to perform forward computation on the graphics processor, obtains the output text tag sequence, decodes it into code text, and returns it to the application. The application interface displays the code completion results.

[0050] Optionally, a model loading request is received, which is used to request the loading of a target model; the resource requirement information of the target model and the hardware resource availability information of the terminal device are obtained, and the hardware resource availability information is evaluated by domain to determine the available resource quantity of each memory domain; based on the resource requirement information and the available resource quantity of each memory domain, the target model is pre-checked, and resource reservation or loading rejection is performed according to the pre-check result; after the pre-check is passed and the resource reservation is completed, a runtime plan is generated, and according to the capability description information of multiple pre-registered heterogeneous inference backends, the target inference backend for running the target model and the target running device corresponding to the target inference backend are determined according to the runtime plan, and the runtime plan is submitted to the target inference backend; the target model is loaded on the target running device through a unified scheduling layer, and a general session identifier decoupled from the backend is generated, and a rollback operation is triggered when loading fails; an inference request is received through the general session identifier, the target inference backend is scheduled to perform inference on the target running device, and invalidation processing is performed on the generated content and an error response is returned when inference fails or the generated result is abnormal.

[0051] Optionally, after receiving a model loading request, the engine first parses the model file path carried in the request, locates the target model file, and extracts structured information such as model parameter size, quantization precision level, and weight tensor shape from the metadata segment in the file header, thereby determining the model size and quantization method. The engine calls the resource query interface provided by the terminal device's operating system to obtain the current free physical memory capacity and free video memory capacity, and performs a domain-based budgeting operation: allocating the free physical memory capacity to the physical memory domain and the free video memory capacity to the video memory domain. If the terminal device uses a unified memory architecture, the unified memory capacity is assigned to both domains simultaneously. Subsequently, the engine calculates the peak resource requirements for the physical memory domain and the video memory domain respectively based on the model size and quantization method. This calculation is completed by querying a preset quantization-resource conversion table, which records the runtime byte overhead per parameter and temporary buffer scaling factor for each domain for different quantization methods. After obtaining the peak requirements for each domain, the engine reads the preset safety margin coefficient and multiplies the peak requirements of each domain by this coefficient to obtain the safety requirements values ​​for the physical memory domain and the video memory domain. The engine compares the security requirement value of each domain with the corresponding available resource quantity. If the security requirement value of any domain exceeds the available resource quantity, the engine constructs a pre-detection failure response. The response payload lists the security requirement value, available resource quantity, and the difference between the two for each domain, returns it to the caller, and terminates the loading process. If all domains meet the conditions, the engine determines that the pre-detection is successful. It then performs resource reservation for each memory domain, locks the virtual address space of the size corresponding to the security requirement value in the physical memory domain through the operating system interface, allocates the corresponding size of the video memory block in the video memory domain, and marks the above resources as occupied by the current session.

[0052] After the pre-check is passed and resources are reserved, the engine enters the runtime plan generation phase. The runtime plan data structure includes the target model identifier, the amount of reserved resources in each memory domain, the model format identifier, and the session identifier. Based on the model format identifier in the runtime plan and the operating system type of the current terminal device, the engine filters out a set of candidate backends that simultaneously meet the requirements of operating system compatibility and model format compatibility from the constructed structured backend registry. Subsequently, the engine obtains the preset performance score of each candidate backend on the target running device type, and performs a weighted score based on the load status of the current instance of each candidate backend. The candidate backend with the highest weighted score is selected as the target inference backend, its bound computing device is determined as the target running device, and the runtime plan is submitted to the target inference backend.

[0053] The scheduling layer executes the loading operation based on the target inference backend's operating mode. If the operating mode is external process mode, the scheduling layer creates a child process through the managed external runtime manager. After the child process starts, it listens for the local address and port allocated by the built-in HTTP service listening engine. The scheduling layer polls the health check endpoint of this HTTP service for its readiness status. After confirming readiness, it sends the model loading instruction in JSON format to the child process via a POST request. The request body includes parameters such as the target model path, the target running device type, and the amount of reserved resources in each domain. After the child process completes model loading, it returns a ready response. The scheduling layer generates a generic session identifier decoupled from the backend implementation, establishes a mapping between this identifier and the child process's HTTP service address, and returns it to the caller. If the child process returns an error or the health check times out during loading, the scheduling layer determines that loading has failed and triggers a rollback operation: it sends a termination instruction to the child process, releases the reserved resources locked in each memory domain in step S30, clears the established session mapping records, and returns a loading failure error code and a description of the failure reason to the caller.

[0054] The caller initiates an inference request using a common session identifier. The scheduling layer uses this identifier to find the corresponding backend context and HTTP service address, constructs a JSON request body, and sends it to the inference endpoint of the target inference backend. After the child process completes inference, it returns a JSON response body in the same format. The scheduling layer parses the response body to extract the generated content. If the HTTP request returns an error status code, connection times out, or the child process is unresponsive during inference, the scheduling layer determines that the inference has failed, invalidates the generated content, discards the returned partial generated results, returns an error response to the caller, and checks the health status of the child process. If the child process is abnormal, a rollback operation is triggered to release session resources. If the inference completes normally but the returned generated content fails format validation or is empty, the scheduling layer also invalidates the generated content, returns a content error response to the caller, and retains the session for retry. Through the complete closed loop of domain-specific resource budgeting and reservation, runtime plan generation and backend matching, route submission after plan readiness, load failure rollback, and generated content invalidation handling in case of inference exceptions, resource over-provisioning due to lack of pre-checking and incorrect routing due to improper backend selection are reduced.

[0055] This embodiment avoids memory overflows and program crashes caused by blind loading by pre-comparing model resource requirements with available hardware resources on the terminal device and only loading when the conditions for operation are met. Simultaneously, by automatically determining the appropriate target inference backend and target running device from multiple registered heterogeneous inference backends based on their capability descriptions, the inference engine can fully utilize the computational advantages of different backends across multiple platforms such as Windows, macOS, Linux, Android, and iOS, overcoming the cross-platform performance loss caused by binding to a single inference framework in related technologies. A unified scheduling layer and a universal session identifier completely shield the differences in backend operating modes, allowing upper-layer applications to complete loading and inference with a consistent interface, significantly reducing client adaptation complexity and ensuring the stability and efficiency of large-scale model inference on the client side.

[0056] Based on any of the above embodiments, in Embodiment 2 of this application, before receiving the model loading request, the process includes: A10, through a cross-platform unified abstraction layer, scans specified paths or parses configuration files on the operating system running on the terminal device to discover multiple available heterogeneous inference backends. A20, obtain the registration information submitted by each of the heterogeneous inference backends, wherein the registration information includes at least the backend identifier, the supported operating system, the operating mode, and the supported computing device type; A30, based on the registration information of each heterogeneous inference backend, construct a structured backend registry as the capability description information of each of the multiple heterogeneous inference backends.

[0057] In this embodiment, the cross-platform unified abstraction layer refers to a layer of encapsulation built on top of the native application programming interfaces of different operating systems. It provides other modules of the engine with platform-independent file system access, dynamic library loading, and process management interfaces. The specified path refers to a preset directory used to store the dynamic library files or executable files of the inference backend. This directory is determined and recorded in the engine's global configuration during engine installation. The configuration file refers to a structured data file containing backend registration information. Its format can be key-value pair text or markup language document, and its storage location can be a specific subdirectory under the engine's installation directory. The registration information is a set of data in which each inference backend declares its capabilities to the engine. It includes at least the backend identifier, supported operating systems, operating modes, and supported computing device types. The backend identifier is a string or enumeration value that uniquely distinguishes each inference backend. The operating mode describes how the backend is invoked by the engine, including an embedded mode that executes directly within the engine process address space and a separate mode that runs as an independent child process. The supported computing device type indicates the type of hardware resources that the backend can schedule, such as a central processing unit, a graphics processing unit, or a neural network processor. The structured backend registry is a data table maintained in memory by the engine. It uses each backend identifier as an index and stores all the registration information of that backend, serving as the basis for the engine's ability to select backends in subsequent operations.

[0058] As an optional implementation, during engine initialization, the engine first traverses all files under a specified path through the file system module in the cross-platform unified abstraction layer. This file system module uses Unicode paths and multi-byte character set conversion encapsulation on Windows platforms and a POSIX-compliant directory traversal interface on Unix-based platforms, exposing a unified iterator object to the upper layer. For each dynamic library file ending with a specific suffix under the specified path, the engine attempts to load and execute its exported standardized registration function. This registration function, implemented by each backend, returns a pre-filled registration information data structure, containing the backend identifier string, the backend's supported operating system identifier bitmask, the running mode enumeration value, and a list of supported computing device types. Upon receiving this structure, the engine extracts each field and inserts it as a record into the backend registry in memory. If a dynamic library file fails to load or its exported registration function returns an error, that backend is silently skipped, without affecting the discovery of other backends. In this way, adding a new backend only requires placing its dynamic library file in the specified path to be automatically discovered and registered, without modifying the engine's main program.

[0059] As an alternative implementation, the engine does not traverse the dynamic library directory during startup. Instead, it parses a backend registration configuration file pre-deployed in the engine configuration directory. This configuration file uses an Extended Markup Language (ESL) format, with multiple backend declaration elements under the root node. Each element contains sub-elements such as backend identifier, operating system compatibility list, runtime mode tag, and computing device support tag. The engine reads this file through a configuration parser in the cross-platform unified abstraction layer. The configuration parser internally encapsulates file reading and memory mapping implementations for different operating systems. After parsing, the engine writes each extracted backend configuration information to the backend registry in memory. When a backend exists as an independent process, its executable file path is also recorded in the corresponding entry in the configuration file. This approach decouples backend discovery from the actual loading of the backend, allowing the engine to collect capabilities without loading any third-party code, thus reducing security risks. Furthermore, the configuration file supports manual editing, facilitating the pre-definition of available backends on embedded devices or platforms without file system dynamic loading capabilities, improving deployment flexibility.

[0060] For example, on a terminal device running a Linux operating system, the user installs this inference engine through a package manager. After the engine is installed, a backend registration configuration file is automatically deployed in the configuration directory. This file uses a structured data format to describe the currently available inference backends. The user starts the inference engine daemon, and during the engine initialization phase, the configuration file is read and parsed through a cross-platform unified abstraction layer. The configuration file declares two inference backends: the first record is identified as "llamacpp", which supports the Linux operating system, runs in external process mode, and its executable file path points to the llama-server binary file in the engine installation directory. Supported computing devices include CPUs and GPUs. The second record is identified as "mnn", which supports Linux and Android operating systems, runs in external process mode, and its executable file path points to the MNN inference service binary file. It supports CPUs, GPUs, and neural network processors. The engine parses these two records one by one and writes them to a structured backend registry in memory. Each record serves as a capability description for that backend. Since each backend is declared as an external process, the engine incorporates it into the managed external runtime management system. When loading models later, the corresponding backend will be started by creating child processes, and its lifecycle will be managed uniformly by the engine. At this point, the engine has completed its backend registration. The structured records in the backend registry will then serve as the capability description information used for backend selection during subsequent model loading.

[0061] This embodiment shields the differences in file access and library loading across different operating systems through a cross-platform unified abstraction layer, enabling the backend discovery logic to run with a single codebase across platforms and reducing maintenance costs. A structured backend registry aggregates the capability descriptions of various heterogeneous backends, providing standardized data for subsequent automatic backend selection. This achieves discoverability and scalability of backend capabilities, allowing the engine to integrate multiple inference frameworks and provide unified services to higher layers.

[0062] Based on any of the above embodiments, in Embodiment 3 of this application, the pre-detection of the target model based on the resource requirement information and the hardware resource availability information includes: Step S31: Extract the model volume and quantization method of the target model from the metadata of the target model.

[0063] Step S32: Determine the peak resource requirements of the target model in each memory domain according to the model volume and the quantization method.

[0064] Step S33: Based on the preset safety margin coefficient, the peak resource requirements of each memory domain are adjusted upwards to obtain the safety requirement value corresponding to each memory domain.

[0065] Step S34: Compare the security requirement value corresponding to each memory domain with the available resources of the corresponding memory domain; if the security requirement value of any memory domain exceeds the available resources of the corresponding memory domain, the pre-check is determined to have failed, the model loading request is rejected, and a prompt message containing the resource gap between each memory domain is output; if the security requirement value of all memory domains does not exceed the available resources of the corresponding memory domain, the pre-check is determined to have passed, and resource reservation is performed for each memory domain.

[0066] In this embodiment, model volume refers to the original space occupied by the weight parameters of the target model on the storage medium. Quantization method refers to the specific method of compressing model weights from their original precision to a lower precision, which determines the actual byte occupancy of a single parameter during inference. Memory domain is a logical partition of different types of memory resources on the terminal device, including at least physical memory domain and video memory domain. The physical memory domain corresponds to the system's main memory, and the video memory domain corresponds to the dedicated storage unit of the graphics processor. Peak resource requirement refers to the estimated maximum amount of resources required by the target model in each memory domain during inference. The safety margin coefficient is a preset floating-point number greater than one, used to independently reserve buffer space for each memory domain based on the peak requirement. The safety requirement value is the final resource requirement after adjusting the coefficient. Available resource amount refers to the currently unoccupied free resource capacity of each memory domain of the terminal device. Resource reservation refers to locking the resource amount corresponding to the safety requirement value required by each memory domain in advance after the pre-check passes, preventing other tasks from competing for it.

[0067] As an optional implementation, the engine reads the metadata segment at a fixed offset in the header of the target model file, parses out the parameter size and quantization precision enumeration values, multiplies the parameter size by the original precision in bytes to obtain the model volume, and records the quantization method identifier. Internally, the engine maintains a domain-specific requirement calculation table, which records the byte overhead per parameter and temporary buffer scaling factor for different quantization methods in the physical memory domain and video memory domain. The engine divides the model volume by the original precision in bytes to obtain the total number of parameters, multiplies this by the byte overhead per parameter in the physical memory domain and the byte overhead per parameter in the video memory domain, then multiplies each by the corresponding domain scaling factor and adds a fixed overhead to obtain the peak resource requirements for the physical memory domain and the video memory domain. Subsequently, the engine reads a preset safety margin coefficient, multiplies the peak requirements of both domains by this coefficient, and obtains the safety requirement values ​​for the physical memory domain and the video memory domain. The engine obtains the current available physical memory and video memory from the resource monitoring module, compares the physical memory domain safety requirement value with the available physical memory, and compares the video memory domain safety requirement value with the available video memory. If any security requirement value exceeds the corresponding available amount, the engine constructs a pre-detection failure response. The response includes the security requirement value and available amount for both the physical memory domain and the video memory domain, as well as the gap value for each domain. If both are satisfied, the engine calls the operating system interface to lock a virtual address space of the size corresponding to the security requirement value in the physical memory domain and mark it as reserved. In the video memory domain, it allocates a video memory block of the corresponding size through the graphics interface and marks it as occupied. After completing the resource reservation, the engine returns a pre-detection pass signal.

[0068] As an alternative implementation, the engine does not use static locking allocation when handling resource reservations. Instead, it generates a resource budget certificate. This certificate records the security requirement value, reservation timestamp, and session identifier for each memory domain and submits it to the operating system or resource management daemon for registration. During subsequent model loading and inference, the resource management daemon monitors and protects the resource usage of each memory domain based on this certificate. When it detects other processes attempting to preempt reserved resources, it protects the allocated resources from being reclaimed through priority scheduling or memory pressure notification mechanisms. Simultaneously, the safety margin coefficient is dynamically adjusted based on the historical memory pressure data of the terminal device. If the device frequently experiences memory reclamation events recently, the safety margin coefficient is automatically increased to enhance the redundancy of resource reservations.

[0069] This embodiment effectively avoids runtime memory overflow caused by ignoring memory fragmentation and system fluctuations by incorporating a safety margin coefficient into demand estimation. Using model size and quantization method as input parameters to determine peak demand allows demand estimation to accurately match actual resource consumption under different quantization strategies, avoiding biases caused by rough estimations based solely on model parameter quantities. When a pre-check fails, a structured prompt message containing the safety demand value and available resources is output, enabling users to intuitively understand resource gaps, reducing time spent on blind attempts and troubleshooting, and improving the deployment efficiency of model loading.

[0070] Based on any of the above embodiments, in Embodiment 4 of this application, referring to Figure 2 Based on the capability descriptions of multiple pre-registered heterogeneous inference backends, a target inference backend for running the target model and a target running device corresponding to the target inference backend are determined, including: Step S41: Extract the operating system, model format, and computing device type of each of the pre-registered heterogeneous inference backends.

[0071] In this embodiment, the capability description information is the set of backend attributes stored in the structured backend registry built during the engine initialization phase. Operating system refers to the platform identifier on which the backend can run normally. Model format refers to the type of model file that the backend can parse and execute. Computing device type refers to the category of hardware units that the backend can schedule to perform computations.

[0072] Optionally, the engine traverses the backend registry in memory and performs field extraction operations on each record in the registry. During extraction, the operating system identifier field, model format list field, and computing device type list field stored in the record are read sequentially using the record index as the key. The extraction results are then organized into a set of filtering tuples using the backend identifier as the index and temporarily stored in the temporary decision context generated after the pre-screening is passed, for use in subsequent filtering steps.

[0073] Step S42: Based on the operating system of the terminal device and the model format of the target model, candidate inference backends are selected from the multiple heterogeneous inference backends.

[0074] In this embodiment, candidate inference backends refer to the set of inference backends that are eligible to participate in subsequent performance scoring after being filtered by two layers of operating system compatibility and model format compatibility.

[0075] Optionally, the engine obtains the operating system type identifier of the current terminal device, which is determined and stored in the global environment variables through a cross-platform unified abstraction layer when the engine starts. It also obtains the model file format identifier determined during the parsing of the target model file in step S10. The engine iterates through each backend tuple extracted in step S41, performing a two-level matching check on each backend: the first level compares the operating system identifier in the backend tuple with the operating system identifier of the terminal device; the second level determines the inclusion relationship between the model format list in the backend tuple and the format identifier of the target model. Backends that pass both levels of checks are marked as candidate inference backends and added to the candidate set. Backends that fail either level of check are directly excluded.

[0076] Step S43: Obtain the performance scores of each candidate inference backend on different computing device types, and combine them with the current available resource balance of the terminal device to perform a weighted score on each candidate inference backend to obtain a backend sequence.

[0077] In this embodiment, the performance score is a pre-calibrated quantitative score reflecting the relative speed of each inference backend in executing standard inference tasks on different computing devices. Current available resource margin refers to the currently idle computing power and memory capacity of each computing device type, including the number of idle cores and available physical memory of the CPU, and the proportion of idle stream processors and available video memory of the GPU.

[0078] Optionally, the engine internally maintains a performance scoring matrix, which stores normalized performance scores obtained from offline benchmark tests, with inference backend identifiers as rows and computing device types as columns. The engine iterates through the candidate inference backend set obtained in step S42, and for each candidate backend, reads its corresponding performance score for each computing device type from the performance scoring matrix. Simultaneously, the engine calls the resource monitoring module to obtain the current available resource balance for each computing device type, mapping the available resource balance to a resource sufficiency factor. The more abundant the resource balance, the closer the factor is to one; when resources are insufficient, the factor approaches zero. The engine multiplies the performance score of each candidate backend on each computing device by the corresponding device's resource sufficiency factor to obtain a weighted score. Finally, the engine sorts the candidate inference backends from highest to lowest weighted score, forming a backend sequence.

[0079] Step S44: Select the candidate inference backend with the highest weighted score from the backend sequence as the target inference backend, and determine the computing device corresponding to the weighted score among the computing device types supported by it as the target running device.

[0080] In this embodiment, the target inference backend is the inference framework ultimately selected by the engine to actually carry out the target model loading and inference tasks. The target runtime device is the physical computing unit that the inference framework will bind to when performing calculations.

[0081] Optionally, the engine directly takes the first element of the backend sequence generated in step S43 as the target inference backend. During the weighted scoring process in step S43, the highest weighted score of each candidate backend also records the type of computing device that generated that score. The engine determines this computing device type as the target running device and writes the identifier of the target inference backend and the type identifier of the target running device together into the decision result structure, which is then passed to the subsequent unified scheduling layer to execute model loading. If the backend sequence is empty, it means that there is no available backend, and the engine returns an error response to the caller.

[0082] This embodiment uses the operating system, model format, and computing device type in the capability description information as multi-level screening conditions. This ensures that the selection process for candidate inference backends simultaneously considers platform compatibility, model format adaptability, and hardware scheduling capabilities, avoiding loading failures or runtime errors caused by assigning models to incompatible or unsuitable backends. By introducing a weighted mechanism that combines the performance scores of each candidate backend on different computing devices with the current available resource balance of the terminal device, the final selected target inference backend and target running device not only possess theoretically high performance but also match the real-time load status of the current device. This overcomes the shortcomings of static selection strategies, which may allocate inference tasks to congested devices when the device load changes dynamically. The quantitative ranking of weighted scores and the highest-score selection mechanism make the entire backend selection process deterministic and reproducible, eliminating the trial-and-error costs of users manually specifying backends and devices. In complex scenarios with diverse model formats, varying backend capabilities, and real-time fluctuations in device status, adaptive optimal scheduling of the edge-side inference engine is achieved.

[0083] Based on any of the above embodiments, in Embodiment 5 of this application, the operating mode of the target inference backend includes an embedded driver mode and an external process mode. The step of loading the target model on the target running device through a unified scheduling layer, based on the operating mode of the target inference backend, and generating a universal session identifier decoupled from the backend implementation, includes: Step S401: When the target inference backend is running in the embedded driver mode, the loading function of the target inference backend is called in the engine process to load the target model into the memory of the target running device and generate the general session identifier that is decoupled from the backend.

[0084] In this embodiment, the embedded-driven mode refers to the target inference backend being directly loaded by the engine process as a dynamic link library or static library. It runs within the engine's process address space, and all functions provided by the backend are executed through native function calls. The loading function is one of the standardized interfaces exposed by the target inference backend, used to parse the model file from persistent storage and map it into the memory of the specified computing device, completing the computation graph construction and weight initialization required for inference.

[0085] Optionally, after obtaining the decision result structure, the engine's unified scheduling layer checks the target inference backend running mode field. When the enumerated value of this field corresponds to an embedded driver mode, the scheduling layer looks up the corresponding module handle in the loaded backend module table based on the backend identifier, and then obtains the loading function pointer exported by that backend. The scheduling layer uses the local file path of the target model and the device identifier of the target running device as parameters, and initiates the call through this function pointer. Inside the backend loading function, operations such as model file parsing, quantization parameter dequantization, computation graph optimization, and memory allocation are performed, creating all weight tensors and intermediate buffers in the memory of the target running device. After loading is complete, the loading function returns a backend-private model context handle. After receiving this handle, the scheduling layer allocates a globally unique integer general session identifier and establishes a mapping relationship between this identifier and the backend context handle and backend type in the internal mapping table. This general session identifier is then returned to the upper-layer application, and all subsequent operations of the application are performed through this identifier.

[0086] Step S402: When the target inference backend is running in the external process mode, a subprocess corresponding to the target inference backend is created, an inter-process communication channel is established between the subprocess and the subprocess, a loading instruction and the path information of the target model are sent to the subprocess through the inter-process communication channel, and after receiving the ready confirmation returned by the subprocess, a general session identifier that is decoupled from the backend is generated.

[0087] In this embodiment, the external process mode refers to the target inference backend existing as an independent executable file. The engine runs this backend by creating child processes, and inference requests and results are transmitted through an inter-process communication channel. The inter-process communication channel is a bidirectional data transmission channel established between the engine process and the child process, and it is established when the child process is created by inheriting file descriptors or named pipes. Readiness confirmation is a response signal sent by the child process to the engine process after successfully loading the target model.

[0088] Optionally, when the scheduling layer determines that the target inference backend is in external process mode based on its running mode field, it retrieves the executable file path corresponding to that backend from the backend registry. The scheduling layer calls the operating system's process creation interface, passing in the executable file path and predefined command-line parameters, to create a child process. During child process creation, the scheduling layer pre-creates a pair of local sockets or anonymous pipes, redirecting one end as the child process's standard input / output, while retaining the other end within the engine process, thus establishing a bidirectional inter-process communication channel. Once the child process starts, it begins listening for control commands on this channel. The scheduling layer constructs a load request message according to the agreed-upon communication protocol, with the message payload containing the complete path to the target model file and parameters such as the target running device type. This message is serialized and sent to the child process through the channel. Upon receiving the message, the child process executes a model loading process similar to that in the embedded driver mode, and upon completion, sends a ready confirmation message back to the engine process. Upon receiving the ready confirmation, the scheduling layer also allocates a globally unique session identifier, establishes a mapping between this identifier and the communication channel and the child process handle, and returns the identifier to the upper-layer application. At this point, the upper-layer application is completely transparent to the model loading process for both operating modes.

[0089] This embodiment unifies the abstraction of two operating modes within the same scheduling layer, enabling the engine to adaptively choose between direct in-process invocation or isolated execution via child processes when loading models. Both modes output a generic session identifier to the caller. The embedded driver mode reduces inter-process communication overhead and is suitable for latency-sensitive interactive inference scenarios; the external process mode confines backend crashes to child processes, avoiding impact on the stability of the engine's main process, and is suitable for deployment environments that require running multiple heterogeneous backends simultaneously or have high reliability requirements. The unified session identifier generation mechanism shields the differences in underlying operating modes, allowing upper-layer applications to transparently use backend services in any operating mode without modifying code, significantly reducing adaptation and maintenance costs.

[0090] Based on any of the above embodiments, in Embodiment Six of this application, when the target inference backend's operating mode is the external process mode, the step of creating a subprocess corresponding to the target inference backend includes: B1. If the terminal device is running Windows, then the child process is assigned to a job object, so that the lifecycle of the child process is bound to the job object. B2. If the terminal device is running a Unix operating system, then the process identifier of the child process is recorded, and a mechanism is set for the child process to receive a termination signal when the engine process exits.

[0091] In this embodiment, a job object is a kernel-level resource management mechanism provided by the Windows operating system. It allows a group of processes to be associated with a unified job unit, and imposes common restrictions and lifecycle controls on all processes within that job unit. When a job object is closed or destroyed, the operating system kernel forcibly terminates all associated processes under that job object. A process identifier is a unique numerical identifier assigned to each process by the kernel in the Unix operating system. The termination signal mechanism refers to an inter-process notification method set through system calls. When the parent process exits abnormally, the kernel or the parent process sends a preset termination signal to the child process, and the child process captures this signal and performs exit processing.

[0092] As an optional implementation, after creating the child process, the engine's process management module executes the corresponding managed policy based on the operating system type of the terminal device. When the operating system type is Windows, the engine pre-creates a job object through the CreateJobObject interface before creating the child process, and suspends the child process by passing the CREATE_SUSPENDED flag when creating the child process. After the child process is successfully created, the engine calls the AssignProcessToJobObject interface to associate the handle of the child process with the job object, and then calls ResumeThread to resume the execution of the child process. From then on, the lifecycle of the child process is bound to the job object. When the engine process exits, the operating system kernel destroys the job object handle during the process of reclaiming engine process resources and terminates all child processes allocated to the job object. When the operating system type is Unix, after creating the child process through fork and exec, the engine calls the setpgid interface to put the child process into a newly created process group, and the group identifier of the process group is set to the process identifier of the child process itself. At the same time, the engine process also adds itself to the process group, or maintains the mapping relationship between the process group identifier and the current session. When the engine process exits, its exit handler sends a SIGTERM signal to the process group, notifying all child processes in the group to perform cleanup and exit. If the child processes still have not exited after the timeout, a SIGKILL signal is sent to the process group, and the operating system kernel terminates all remaining child processes in the process group.

[0093] As an alternative implementation, on the Windows platform, the engine encapsulates the creation of job objects and process association within a single process-managed factory class. This factory class instantiates a global job object upon engine startup and sets the job object's `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` attribute to enabled. Whenever a child process is created via external process mode, the factory class allocates the job object during the atomic process creation operation, eliminating the need for separate calls to the `AssignProcessToJobObject` interface each time. This attribute ensures that if the job object handle held by the engine process is closed for any reason, including normal engine process exit, crash, or forced termination, all child processes under the job object are synchronously terminated by the system, avoiding child process remnants caused by interrupted code execution flow due to abnormal engine crashes or unexecuted cleanup logic. On the Unix platform, in addition to setting the `PR_SET_PDEATHSIG` attribute, the engine places all child processes into an independent process group and monitors their exit status in the engine process's main event loop by polling with `waitpid` or registering a `SIGCHLD` signal handler. When the engine process is about to exit, its exit handler sends a SIGTERM signal to the entire child process group, waiting for all child processes to exit voluntarily within the timeout window. After the timeout, a SIGKILL signal is sent to force cleanup, ensuring that no child process escapes the engine's control and becomes an orphan process.

[0094] This embodiment achieves a unified child process management strategy across platforms by introducing job objects to bind the lifecycle of child processes at the kernel level on the Windows platform and setting a termination signal mechanism when the parent process exits on the Unix platform. Regardless of whether the engine process exits normally or crashes abnormally, the operating system can automatically reclaim all backend child processes, completely resolving the problem of backend process remnants caused by signal handling oversights or abnormal code execution paths in related technologies. This design, which transfers process cleanup responsibility from user-mode code to the operating system kernel mechanism, significantly improves the stability of the inference engine and the reliability of resource reclamation under long-term operation and abnormal scenarios.

[0095] Based on any of the above embodiments, in Embodiment 7 of this application, after creating the subprocess corresponding to the target inference backend, the process includes: C1. If the engine process exits abnormally and the operating system running on the terminal device is Windows, the operating system destroys the job object corresponding to the engine process, and terminates all child processes assigned to the job object by destroying the job object. C2. If the engine process exits abnormally and the terminal device is running a Unix operating system, the child process will detect the exit of the engine process through the termination signal mechanism and terminate itself according to the termination signal mechanism.

[0096] In this embodiment, abnormal engine process exit refers to the situation where the engine main process terminates directly without performing the normal exit cleanup process due to reasons such as uncaught abnormal signals, memory access violations, forced termination by the operating system, or fatal errors within the process. Operating system destruction of job objects refers to the Windows kernel automatically reclaiming all kernel objects created and held by a process when it terminates, including job object handles. Terminating all child processes assigned to a job object means that during the job object destruction process, the kernel traverses all associated processes under that job object and forcibly sends termination commands. This behavior is guaranteed by the kernel and does not depend on any user-mode cleanup code.

[0097] Optionally, when the engine process runs on the Windows operating system, the engine creates a job object using `CreateJobObject` during the startup phase. Each time a child process is created in external process mode, the child process is associated with this job object using `AssignProcessToJobObject`. The job object itself has a handle held by the engine process, and the handle's lifecycle is bound to the engine process. When the engine process exits abnormally—whether due to an application crash, a user forcibly terminating the process via Task Manager, or process termination triggered by the system shutdown process—the Windows kernel closes all kernel handles opened by the process, including the job object handle, during the process of reclaiming the engine process's resources. Since the `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` flag is set when the job object is created, the kernel triggers the job object's destruction process when closing the last handle of the job object. During the destruction process, the kernel traverses the list of associated processes maintained internally by the job object and performs a `TerminateProcess` operation on all child processes assigned to the job object, forcibly terminating their execution. The entire process is completed in kernel mode, unaffected by whether the user-mode code of the engine process executes normally, thus ensuring that no backend subprocesses are left behind even in the most extreme crash scenarios.

[0098] Optionally, when the engine process runs on a Unix operating system, after creating a child process via fork, the engine sets the PR_SET_PDEATHSIG attribute for the child process using the prctl system call, specifying that the parent process should send a SIGTERM signal to the child process upon exiting. Simultaneously, the child process also registers a handler function for this signal. When the engine process exits abnormally, the kernel marks the engine process as terminated, identifies all its child processes, and sends the termination signal preset by PR_SET_PDEATHSIG to each child process. Upon receiving the SIGTERM signal, the signal handler function is triggered, and the child process begins executing its own cleanup logic, including releasing memory or video memory occupied by loaded model weights, closing open file descriptors and network connections, and then actively exiting by calling the exit system call. If the child process fails to respond to the SIGTERM signal and exit within the preset time due to uninterruptible kernel operations or deadlock, the timeout monitoring mechanism that was synchronously started when the termination signal mechanism was set will come into play. This monitoring mechanism is implemented through a separate watchdog process or system timer. When a child process is detected to be still alive after a timeout threshold, a SIGKILL signal is sent directly to it. This signal cannot be caught or ignored, and the kernel will forcibly terminate the child process.

[0099] This embodiment completely eliminates the lack of cleanup code execution paths caused by engine process anomalies by shifting the core logic of process cleanup from passive waiting in user mode to active triggering by the operating system kernel. On the Windows platform, the kernel-level forced termination of associated processes when job objects are destroyed enables synchronous reclamation of downstream subprocesses. On the Unix platform, the PR_SET_PDEATHSIG signal mechanism is combined with a watchdog strategy of forced termination due to timeout, giving child processes an opportunity to clean up resources in an orderly manner while ensuring that no orphan processes are left behind in extreme cases. Neither mechanism on these platforms relies on writing flawless exit handling code in the engine process, fundamentally solving the problem of process residue and resource leakage caused by occasional crashes during long-term deployment of the client-side inference engine, significantly improving the long-term operational stability of the system.

[0100] Based on any of the above embodiments, in Embodiment 8 of this application, receiving an inference request through the general session identifier, scheduling the target inference backend to perform inference on the target running device, and returning the inference result includes: Step S501: Based on the general session identifier, find the backend context corresponding to the general session identifier. The backend context includes at least the running mode of the target inference backend, and when the running mode is external process mode, the backend context also includes a communication channel.

[0101] In this embodiment, the backend context is an internal data structure created by the engine for each generic session identifier during the model loading phase. It stores all backend-related information associated with that session. This data structure is created synchronously and written to the mapping table when the generic session identifier is generated in step S40.

[0102] Optionally, when the engine receives an inference request, it first extracts a generic session identifier from the request parameters. The scheduling layer uses this identifier as the key to perform a hash lookup operation in its maintained global session mapping table. This mapping table uses an open-chaining method to resolve hash collisions, and each entry contains a generic session identifier, a backend context pointer, and a session status flag. After a successful lookup, the scheduling layer obtains the corresponding backend context pointer and reads the target inference backend's runtime mode enumeration field and communication channel file descriptor from this context. The runtime mode enumeration field indicates whether the backend corresponding to this session is in embedded driver mode or external process mode. In external process mode, the communication channel points to the socket connection established with the child process; in embedded driver mode, this field is null.

[0103] Step S502: When the running mode is the embedded driver mode, the inference interface of the target inference backend is called to obtain the inference result.

[0104] In this embodiment, the inference interface is a standardized function exposed by the target inference backend, whose function signature accepts the model context handle, the input tensor data pointer, and the output tensor buffer pointer as parameters.

[0105] Optionally, the scheduling layer checks the runtime mode enumeration field in the backend context. When it determines that the mode is embedded, it retrieves the backend private model context handle saved during the model loading phase from the backend context. The scheduling layer rearranges the input data passed in by the caller in memory according to the tensor layout required by the backend, constructing an input tensor descriptor containing a data pointer, a dimension array, and element types. Subsequently, the scheduling layer calls the backend's inference interface function, passing in the model context handle, the input tensor descriptor, and a pre-allocated output buffer pointer. During execution, the inference interface feeds the input tensor into the computation graph for forward propagation, and writes the computation result to the output buffer. After the call returns, the scheduling layer reads the output tensor data from the output buffer and records the shape and element types of the output tensor as metadata of the inference result.

[0106] Step S503: When the running mode is external process mode, the inference request is serialized and sent to the subprocess corresponding to the target inference backend through the communication channel, and the inference result returned by the subprocess is received.

[0107] In this embodiment, inference request serialization refers to the process of encoding input tensor data and request metadata into a byte stream according to a communication protocol. In external process mode, the communication channel is the bidirectional data transmission channel between the engine process and the child process established in step S402.

[0108] Optionally, the scheduling layer checks the run mode enumeration field in the backend context. When it determines that it is in external process mode, it reads the corresponding HTTP service address and port number from the backend context. This address and port number are assigned by the engine and passed to the child process via command-line parameters when the child process starts. The child process internally starts a lightweight HTTP service to listen on this address. Before sending the inference request, the scheduling layer first sends a GET request to the health check endpoint of the HTTP service. If a status code of 200 is returned, the service is confirmed to be ready. After the health check is passed, the scheduling layer constructs a JSON request body according to the OpenAI-compatible Chat Completion request format for the input text passed by the caller. The request body includes a model field set to the target model identifier, a messages field encapsulating the user input content, and a stream field set to false to indicate a non-streaming return. The scheduling layer sends this JSON request body to the inference endpoint of the child process's HTTP service via an HTTP client using the POST method. After the child process completes inference, it returns a JSON response body in the OpenAI-compatible Chat Completion response format. The scheduling layer parses the response body and extracts the content field value of the message object from the first element of the choices array as the inference result text. If the stream field in the request is set to true, the child process returns the generated content in SSE format chunk by chunk. The scheduling layer parses the SSE event stream line by line, splices the generated content chunk by chunk, and finally summarizes it into a complete inference result.

[0109] Step S504: Return the obtained inference results to the caller in a unified data format.

[0110] In this embodiment, the unified data format refers to a structured output format defined by the engine that is independent of the backend, which includes at least an output tensor data buffer, a tensor shape array, and a data type identifier.

[0111] Optionally, after obtaining the inference result in step S502 or S503, the scheduling layer converts it into a unified output data structure, regardless of whether it comes from a direct call in the embedded driver mode or a channel reception in the external process mode. This structure contains a pointer to a contiguous memory block storing all element values ​​of the output tensor in row-major order; an integer array recording the size of each dimension of the output tensor; and an enumeration value identifying the data type of the output tensor elements, such as 32-bit floating-point or 64-bit integer. The scheduling layer fills the pointer to this unified structure into the response buffer passed in by the caller and sets the return status code to success. Upon receiving the response, the caller parses the output tensor based on the data type identifier and shape array in the structure, without needing to concern itself with the specific implementation differences of the underlying inference backend.

[0112] This embodiment achieves unified scheduling and distribution of inference requests by finding the backend context and identifying the running mode based on a common session identifier during the inference execution phase. For the embedded driver mode, direct function calls are used to obtain inference results, reducing intermediate data copying and serialization overhead and ensuring fast response times in low-latency inference scenarios. For the external process mode, structured serialization and communication channel transmission are used to isolate the engine main process from the backend inference computation, ensuring that the crash or anomaly of the backend process will not affect the overall availability of the engine service. Finally, the inference results obtained in both modes are unified into a backend-independent data format and returned to the caller, enabling upper-layer applications to interact with a completely consistent interface and data structure. This eliminates the interface adaptation work caused by backend switching and achieves complete transparency of the inference process in complex scenarios with multiple backends deployed collaboratively on the client side.

[0113] This application provides a control device for a cross-platform edge-side large model inference engine. The control device for the cross-platform edge-side large model inference engine includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the control method for the cross-platform edge-side large model inference engine in the first embodiment described above.

[0114] The following is for reference. Figure 3 This document illustrates a schematic diagram of a control device suitable for implementing the edge-side cross-platform large model inference engine of the embodiments of this application. The control device for the edge-side cross-platform large model inference engine in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablets, and in-vehicle terminals, as well as fixed terminals such as digital TVs and desktop computers. Figure 3The control device for the edge-side cross-platform large model inference engine shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0115] like Figure 3 As shown, the control device of the edge-side cross-platform large model inference engine may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1002 or programs loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the control device of the edge-side cross-platform large model inference engine. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the control device of the edge-side cross-platform large model inference engine to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a control device for an edge-side cross-platform large model inference engine with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0116] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0117] The control device for a cross-platform large model inference engine provided in this application adopts the control method for a cross-platform large model inference engine in the above embodiments, which can solve the technical problem of performance loss when the model runs across platforms. Compared with the prior art, the beneficial effects of the control device for a cross-platform large model inference engine provided in this application are the same as those of the control device for a cross-platform large model inference engine provided in the above embodiments, and other technical features in the control device for a cross-platform large model inference engine are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0118] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0120] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the control method of the edge-side cross-platform large model inference engine in the above embodiments.

[0121] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), or any suitable combination thereof.

[0122] The aforementioned computer-readable storage medium may be included in the control device of the edge-side cross-platform large model inference engine; or it may exist independently and not be assembled into the control device of the edge-side cross-platform large model inference engine.

[0123] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the control device of the edge-side cross-platform large model inference engine, the control device of the edge-side cross-platform large model inference engine causes the following: It receives a model loading request, which requests the loading of a target model; it obtains the resource requirement information of the target model and the hardware resource availability information of the terminal device; based on the resource requirement information and the hardware resource availability information, it performs a pre-check on the target model; if the pre-check passes, it determines the target inference backend for running the target model and the target running device corresponding to the target inference backend based on the capability description information of multiple pre-registered heterogeneous inference backends; through a unified scheduling layer, based on the running mode of the target inference backend, it loads the target model on the target running device and generates a universal session identifier decoupled from the backend; it receives inference requests through the universal session identifier, schedules the target inference backend to perform inference on the target running device, and returns the inference result.

[0124] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0126] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0127] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the control method of the aforementioned cross-platform large model inference engine on the edge, thereby solving the technical problem of performance loss when the model runs across platforms. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the control method of the cross-platform large model inference engine on the edge provided in the above embodiments, and will not be repeated here.

[0128] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the control method for a cross-platform large model inference engine on the edge as described above.

[0129] The computer program product provided in this application can solve the technical problem of performance loss when the model runs across platforms. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the control method of the edge-side cross-platform large model inference engine provided in the above embodiments, and will not be repeated here.

[0130] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A control method for a cross-platform, large-model inference engine on the edge, characterized in that, The control method for the edge-side cross-platform large model inference engine includes: Receive a model loading request, which is used to request the loading of the target model; Obtain the resource requirement information of the target model and the hardware resource availability information of the terminal device, perform domain-based budgeting on the hardware resource availability information, and determine the available resource quantity of each memory domain; Based on the resource requirement information and the available resources of each memory domain, the target model is pre-tested; Upon passing the pre-inspection, the target inference backend for running the target model and the target running device corresponding to the target inference backend are determined based on the capability description information of the multiple pre-registered heterogeneous inference backends. Through a unified scheduling layer, based on the operating mode of the target inference backend, the target model is loaded on the target running device, and a general session identifier decoupled from the backend implementation is generated. The inference request is received through the general session identifier, the target inference backend is scheduled to perform inference on the target running device and return the inference result.

2. The control method for a cross-platform large model inference engine on the edge as described in claim 1, characterized in that, Before receiving the model loading request, the method further includes: By using a cross-platform unified abstraction layer, multiple heterogeneous inference backends can be discovered by scanning specified paths or parsing configuration files on the operating system running on the terminal device. Obtain the registration information submitted by each of the heterogeneous inference backends, wherein the registration information includes at least the backend identifier, the supported operating system, the operating mode, and the supported computing device type; Based on the registration information of each heterogeneous inference backend, a structured backend registry is constructed as the capability description information of each of the multiple heterogeneous inference backends.

3. The control method for a cross-platform large model inference engine on the edge as described in claim 1, characterized in that, The pre-detection of the target model based on the resource requirement information and the available resource quantity of each memory domain includes: Extract the model volume and quantization method of the target model from its metadata; Based on the model volume and the quantization method, the peak resource requirements of the target model in each memory domain are determined respectively. Based on the preset safety margin coefficient, the peak resource requirements of each memory domain are adjusted upwards to obtain the safety requirement value corresponding to each memory domain. The security requirement value corresponding to each memory domain is compared with the available resource quantity of the corresponding memory domain. If the security requirement value of any memory domain exceeds the available resources of the corresponding memory domain, the pre-check is determined to have failed, the model loading request is rejected, and a prompt message containing the resource gap between each memory domain is output. If the security requirements of all memory domains do not exceed the available resources of the corresponding memory domain, the pre-check is deemed successful, and resource reservation is performed for each memory domain.

4. The control method for a cross-platform large model inference engine on the edge as described in claim 1, characterized in that, The step of determining the target inference backend for running the target model and the target running device corresponding to the target inference backend based on the capability description information of multiple pre-registered heterogeneous inference backends includes: Based on the capability description information of each of the pre-registered heterogeneous inference backends, the operating system, model format, and computing device type of each heterogeneous inference backend are extracted; Based on the operating system of the terminal device and the model format of the target model, candidate inference backends are selected from the multiple heterogeneous inference backends; The performance scores of each candidate inference backend on different computing device types are obtained, and the weighted scores of each candidate inference backend are calculated in combination with the current available resource balance of the terminal device to obtain the backend sequence. The candidate inference backend with the highest weighted score is selected from the backend sequence as the target inference backend, and the computing device corresponding to the weighted score among the computing device types supported by it is determined as the target running device.

5. The control method for a cross-platform large model inference engine on the edge as described in claim 1, characterized in that, The target inference backend's operating modes include an embedded driver mode and an external process mode. Through a unified scheduling layer, based on the target inference backend's operating mode, the target model is loaded on the target running device, and a generic session identifier decoupled from the backend implementation is generated, including: When the target inference backend is running in the embedded driver mode, the loading function of the target inference backend is called in the engine process to load the target model into the memory of the target running device and generate the general session identifier that is decoupled from the backend implementation. When the target inference backend is running in the external process mode, a child process corresponding to the target inference backend is created, an inter-process communication channel is established between the child process and the child process, a loading instruction and the path information of the target model are sent to the child process through the inter-process communication channel, and after receiving the ready confirmation returned by the child process, a general session identifier that is decoupled from the backend is generated.

6. The control method for a cross-platform large model inference engine on the edge as described in claim 5, characterized in that, When the target inference backend is running in the external process mode, the step of creating a child process corresponding to the target inference backend includes: If the terminal device is running Windows, the child process is assigned to a job object, so that the lifecycle of the child process is bound to the job object; If the terminal device is running a Unix operating system, then the process identifier of the child process is recorded, and a mechanism is set for the child process to receive a termination signal when the engine process exits.

7. The control method for a cross-platform large model inference engine on the edge as described in claim 5, characterized in that, After creating the subprocess corresponding to the target inference backend, the process includes: If the engine process exits abnormally, and the terminal device is running Windows, the operating system will destroy the job object corresponding to the engine process, and terminate all child processes assigned to the job object by destroying the job object. If the engine process exits abnormally and the terminal device is running a Unix operating system, the child process will detect the engine process's exit through a termination signal mechanism and terminate itself according to the termination signal mechanism.

8. The control method for a cross-platform large model inference engine on the edge as described in claim 1, characterized in that, The step of receiving an inference request via the universal session identifier, scheduling the target inference backend to perform inference on the target running device, and returning the inference result includes: Based on the general session identifier, the backend context corresponding to the general session identifier is found. The backend context includes at least the running mode of the target inference backend, and when the running mode is external process mode, the backend context also includes a communication channel. When the operating mode is the embedded driver mode, the inference interface of the target inference backend is called to obtain the inference result; When the running mode is external process mode, the inference request is serialized and sent to the subprocess corresponding to the target inference backend through the communication channel, and the inference result returned by the subprocess is received. The obtained inference results will be returned to the caller in a unified data format.

9. A control device for a cross-platform, large-model inference engine, characterized in that, The control device of the edge-side cross-platform large model inference engine includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the control method for the edge-side cross-platform large model inference engine as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the control method for the edge-side cross-platform large model inference engine as described in any one of claims 1 to 8.

Images