A method for AI model offloading in a desktop cloud scenario

Abstract

This invention discloses a method for unloading AI models in a desktop cloud scenario. The method includes: when an application in a virtual machine initiates a call request to a local AI model, a model request interception and forwarding module captures and encapsulates the call request, generates an unloading task package, and sends it to the edge-side model execution platform; the edge-side model execution platform parses the unloading task package, determines the corresponding executable model, and performs model inference calculations; the model inference calculation results are compressed, encrypted, and sent to the model request interception and forwarding module; the model request interception and forwarding module decrypts the result and injects it into the application, completing the entire AI model call process. This invention provides a method for unloading AI models in a desktop cloud scenario, which can effectively solve the problem that desktop cloud environments lack effective support for AI acceleration resources such as GPUs in 2D virtual desktop scenarios, resulting in inefficient execution of edge-side AI models.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method for unloading AI models in a desktop cloud scenario. Background Technology

[0002] With the deepening development of artificial intelligence technology, AI applications are increasingly demanding real-time response, data privacy, and energy efficiency optimization, driving the evolution of computing architecture from a purely cloud-based model to an "edge-cloud collaborative" model. In the field of personal computing, AI PCs (AIPCs) significantly improve user experience and system autonomy by integrating dedicated AI acceleration hardware with local model inference capabilities. Desktop cloud, as an important evolution of traditional PCs, relies on virtualization platforms (VMPs) to achieve multi-tenant, centralized management computing services and is widely used in office, education, and other scenarios. However, against the backdrop of the rapid popularization of AI-native applications, desktop cloud architecture has exposed serious computing power adaptation deficiencies, especially in 2D virtual desktop scenarios where there is a general lack of effective support for AI acceleration resources such as GPUs, making it difficult to support the deployment and execution needs of edge AI models.

[0003] In desktop cloud environments, AI applications typically rely on virtual machines (VMs) to run applications (APPs), and the local AI models invoked by the APPs need to complete inference tasks through underlying computing power interfaces (such as...). Figure 1 (As shown). Ideally, this process should be efficiently handled by a dedicated AI acceleration unit provided by the host or client. However, in existing desktop cloud solutions, 2D scenes cannot directly access the physical GPU due to the isolation and abstraction of graphics card resources by the virtualization layer. Even in scenarios that support 3D graphics acceleration, GPU partitioning technology is limited by high hardware costs, scarce available devices, and coarse allocation granularity, making it difficult to meet the dynamic requirements of diverse AI models for low latency and high throughput computing power. This prevents many lightweight but frequently accessed edge AI applications from running effectively in desktop cloud environments, severely restricting their applicability and competitiveness in the AI ​​era.

[0004] In existing technologies, virtual machines lack intelligent scheduling and offloading mechanisms for AI model requests. All model calls are forcibly restricted to the VM's own resources, failing to flexibly utilize more efficient external computing nodes. Furthermore, current virtualization platforms do not provide standardized model request interception and redirection capabilities, preventing seamless integration of edge model platforms (regardless of whether deployed on the host machine, other virtual machines, clients, or dedicated computing servers) into virtual desktop workflows. Moreover, due to heterogeneous model vendors and inconsistent interface protocols, existing solutions lack a universal offloading adaptation layer, making it difficult to achieve cross-platform and cross-architecture model compatibility and result feedback. Therefore, in desktop cloud scenarios, how to dynamically offload AI model requests to the optimal computing node in a low-cost and highly compatible manner, while ensuring accurate return of results to virtual applications, has become a critical technical challenge that urgently needs to be addressed. Summary of the Invention

[0005] This invention provides a method for unloading AI models in a desktop cloud scenario, which can effectively solve the technical problems mentioned in the background art, such as the lack of effective support for AI acceleration resources such as GPUs in 2D virtual desktop scenarios in desktop cloud environments, resulting in the inefficient execution of edge AI models, and the inability of existing virtualization architectures to achieve dynamic unloading of model requests and return of results.

[0006] To achieve the above objectives, this invention provides a method for unloading AI models in a desktop cloud scenario, applied to a virtual machine. The virtual machine is equipped with a model request interception and forwarding module, which is communicatively connected to an external edge-side model execution platform. The method includes:

[0007] When an application in the virtual machine initiates a call request to the local AI model, the model request interception and forwarding module captures the call request and suspends its execution process on the virtual machine.

[0008] The captured call request is encapsulated to generate an unload task package and sent to the client-side model execution platform;

[0009] The edge model execution platform receives and parses the unloading task package, determines the corresponding executable model, and performs model inference calculation.

[0010] The results of the model inference calculation are compressed, encrypted, and sent to the model request interception and forwarding module that initiated the request;

[0011] The model request interception and forwarding module receives the compressed and encrypted result of the model inference calculation, decrypts the result of the model inference calculation, and injects it into the application, so that the application can continue to execute subsequent logic and complete the entire AI model calling process.

[0012] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the model request interception and forwarding module is deployed in the kernel layer or user-mode application layer of the operating system in the virtual machine, and achieves comprehensive capture of the AI ​​model loading and inference interface through dynamic link library injection, system call table hijacking or container-level API interception mechanism.

[0013] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the edge model execution platform can be deployed in an independent container environment of the host machine, other virtual machines, client devices, edge computing nodes, or dedicated AI computing power servers. The edge model execution platform selects the optimal execution node according to the load balancing strategy and the network latency evaluation model.

[0014] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the encapsulation includes mapping different vendors' private API call formats to a universal model task description language, wherein the description language includes model hash value, input tensor dimension, data type, quantization method and expected latency threshold.

[0015] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the unloading task package is sent to the end-side model execution platform via a cross-virtual machine communication channel provided by the virtualization platform or a secure data channel between the host machine and the client.

[0016] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, after receiving the unloading task package, the edge model execution platform parses the model identifier to determine the corresponding executable model instance, and then uses the dedicated AI acceleration hardware integrated by the edge model execution platform to complete the model inference calculation; if the model is not deployed locally, a compatible version is automatically downloaded and loaded through a preset model repository, and then the dedicated AI acceleration hardware integrated by the edge model execution platform is used to complete the model inference calculation.

[0017] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the model identifier adopts a globally unique model fingerprint, which is jointly generated by the model structure hash, weight parameter hash and version number; the terminal model execution platform matches the deployed model through the local model cache index, and if no match is found, it triggers the download from the preset model repository. The download process uses a differential update algorithm to transmit only the changed parts.

[0018] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the dedicated AI acceleration hardware includes a neural network processing unit, a tensor computation unit, or a field-programmable gate array, and the edge model execution platform automatically selects the optimal hardware backend to execute the inference task based on the computation graph characteristics of the model.

[0019] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, the results of the model inference calculation are compressed and encrypted, a task identifier and an integrity verification code are attached, and the result is encapsulated into a response message. The response message is then returned to the interception and forwarding module that initiated the request via the original communication path or an independent high-speed backhaul channel.

[0020] In a preferred embodiment of the method for unloading AI models in a desktop cloud scenario provided by the present invention, a model scheduling manager is further deployed at the host machine level. The model scheduling manager monitors the computing power utilization, memory usage, and network status of each edge model execution platform in real time, dynamically allocates target nodes for model unloading based on the priority and service quality requirements of AI tasks, and establishes a model compatibility database to record the performance, accuracy deviation, and resource consumption characteristics of different models on various execution platforms. During scheduling, the optimal execution environment that meets the application requirements is selected by referring to the model compatibility database.

[0021] Compared to existing technologies, the AI ​​model offloading method in desktop cloud scenarios provided by this invention has the following beneficial effects: By setting up a model request interception and forwarding module inside the virtual machine, transparent offloading of AI model calls in the desktop cloud environment is achieved, breaking through the dependence of traditional virtualization architecture on physical AI acceleration resources, enabling 2D virtual desktops to efficiently execute local AI model tasks; by building a standardized task encapsulation and cross-platform communication mechanism, it supports the compatible access of various heterogeneous model frameworks and execution environments, improving the system's flexibility and scalability; the edge-side model execution platform can be flexibly deployed on the host machine, other virtual machines, clients, or dedicated computing power nodes, making full use of distributed computing power resources and significantly reducing deployment and maintenance costs; the task scheduling and result feedback process has integrity verification and encryption protection, ensuring data security and service reliability; the entire solution does not require modification of application code, is compatible with the existing AI ecosystem, has good feasibility and industrial promotion value, and effectively enhances the adaptability and market competitiveness of desktop cloud in the era of artificial intelligence. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0023] Figure 1 This is a diagram illustrating the usage structure of AIPC in existing technologies;

[0024] Figure 2 This is a flowchart of the method for unloading AI models in a desktop cloud scenario provided in this embodiment of the invention;

[0025] Figure 3 This is a usage structure diagram of the method for unloading AI models in a desktop cloud scenario provided in this embodiment of the invention;

[0026] Figure 4 This is a flowchart illustrating the process of unloading an AI model in a desktop cloud scenario, as provided in this embodiment of the invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as superior or more advantageous than other embodiments or designs. Rather, the use of the term "exemplary" is intended to present the concept in a specific manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0029] Currently, AI applications in desktop cloud environments typically rely on virtual machines to run applications, and the local AI models called by the app need to complete inference tasks through underlying computing power interfaces. Ideally, this process should be efficiently handled by a dedicated AI acceleration unit provided by the host machine or client. However, in existing desktop cloud solutions, 2D scenes cannot directly access physical GPUs due to the isolation and abstraction of graphics card resources by the virtualization layer. Even in scenarios that support 3D graphics acceleration, GPU partitioning technology is limited by high hardware costs, scarce available devices, and coarse allocation granularity, making it difficult to meet the dynamic requirements of diverse AI models for low latency and high throughput computing power. This prevents many lightweight but frequently called edge AI applications from running effectively in desktop cloud environments, severely restricting their applicability and competitiveness in the AI ​​era. To address the above technical problems, this invention proposes a method for AI model unloading in desktop cloud scenarios, which can effectively solve the aforementioned technical problems of desktop cloud environments lacking effective support for AI acceleration resources such as GPUs in 2D virtual desktop scenarios, resulting in inefficient execution of edge AI models, and the inability of existing virtualization architectures to achieve dynamic unloading of model requests and result feedback.

[0030] Please refer to the following: Figure 2 , Figure 3 and Figure 4 , Figure 2 This is a flowchart of the method for unloading AI models in a desktop cloud scenario provided in this embodiment of the invention; Figure 3 This is a usage structure diagram of the method for unloading AI models in a desktop cloud scenario provided in this embodiment of the invention; Figure 4 This is a flowchart illustrating the process of unloading an AI model in a desktop cloud scenario according to an embodiment of the present invention. The method for unloading an AI model in a desktop cloud scenario is applied to a virtual machine, which is equipped with a model request interception and forwarding module. This module communicates with an external edge-side model execution platform. The method specifically includes the following steps:

[0031] Step S1: When an application in the virtual machine initiates a call request to the local AI model, the model request interception and forwarding module captures the call request and suspends its execution process on the virtual machine.

[0032] Specifically, when an application running in a virtual machine initiates a call request to a local AI model, the interception and forwarding module captures the call request by hooking the application programming interface or monitoring system calls, and suspends its execution process on the virtual machine.

[0033] The model request interception and forwarding module is deployed in the kernel layer or user-space application layer of the operating system within the virtual machine. It achieves comprehensive capture of AI model loading and inference interfaces through dynamic link library injection, system call table hijacking, or container-level API interception mechanisms. For model calls based on mainstream lightweight inference frameworks such as TensorFlow Lite, PyTorch Mobile, and ONNX Runtime, the module can identify and intercept core inference function calls such as interpreter.run(), model.forward(), or session.run(). After the interception action is triggered, the execution thread within the virtual machine is suspended, and its context state (including register values, stack pointer, and memory mapping) is saved to the kernel buffer for restoration during subsequent result injection. The deployment of the model request interception and forwarding module is independent of specific virtualization technologies and can be implemented in KVM, Xen, Hyper-V, and containerized virtualization architectures through corresponding guest tools or operating system kernel modules. In a Windows environment, it can be implemented through Windows Filtering. API hooks are implemented using Platform (WFP) or Detours libraries; in Linux environments, shared libraries are injected using eBPF programs or the LD_PRELOAD mechanism, thereby transparently intercepting model calls in user space.

[0034] Step S2: Encapsulate the captured call request, generate an unloading task package, and send it to the end-side model execution platform;

[0035] Specifically, the captured call request is standardized and encapsulated, and the model identifier, input data, call parameters and target accuracy requirements in the call request are extracted to generate a structured unloading task package. The unloading task package is then sent to the edge model execution platform through the cross-virtual machine communication channel provided by the virtualization platform or the secure data channel between the host machine and the client.

[0036] The encapsulation includes protocol conversion of the call request, unifying the private API call formats of different vendors into a universal model task description language. This description language can be defined using JSON Schema, containing fields such as: model_fingerprint (model hash value), input_shape (input tensor dimension, e.g., [1,224,224,3]), data_type (data type, e.g., float32, int8), quantization_scheme (quantization method, e.g., symmetric, asymmetric), and latency_budget_ms (expected latency threshold in milliseconds). Input data is appended to the end of the task package in binary Blob form and transferred from the application address space to the task buffer via memory copying. The serialization of the task package uses Protocol Buffers format to ensure cross-language compatibility and parsing efficiency.

[0037] Regarding the choice of communication channel, if the edge-side model execution platform is deployed on the host machine or other virtual machines, a cross-virtual machine communication channel is used. This channel employs a shared memory mapping combined with an event notification mechanism. The host machine configures a dedicated memory area for multiple virtual machines to share. Each virtual machine accesses this area through memory page mapping and synchronizes task delivery and result readiness through interrupt signals or polling flags. Specifically, the host machine exposes a pre-allocated shared memory page (e.g., 4KB in size) to the virtual machine through the VirtIO-MMIO or Hyper-VVMBus interface. The virtual machine maps this page to its own address space using the mmap system call. When a task is delivered, the model request interception and forwarding module writes the serialized task package to the designated slot in the shared memory and notifies the host machine by writing to the I / O port or triggering an MSI-X interrupt. After the host machine's daemon listens for the event, it immediately reads the task package from the shared memory and distributes it to the execution platform. This mechanism has a communication latency of less than 0.5 milliseconds, making it suitable for interactive AI applications with extremely high real-time requirements.

[0038] If the endpoint model execution platform is located on the client device, a secure data channel is used between the host and the client. This channel establishes an encrypted tunnel based on the TLS 1.3 protocol, combined with two-way authentication of device certificates and a dynamic session key update mechanism to prevent model request and result data from being stolen or tampered with during transmission. Before the channel is established, the host and client need to exchange X.509 digital certificates in advance and verify the validity of the certificates through OCSP Stapling. The session key is generated using the ECDHE key exchange algorithm, and a new temporary key pair is used for each session to achieve forward confidentiality. The channel throughput is not less than 1Gbps, which is sufficient to carry model input data of high-resolution images or video streams. Data transmission uses TCP long connection multiplexing to avoid the overhead caused by frequent handshakes.

[0039] Step S3: The edge model execution platform receives and parses the unloading task package, determines the corresponding executable model, and performs model inference calculation.

[0040] Specifically, after receiving the unloading task package, the edge model execution platform parses the model identifier to determine the corresponding executable model instance, and then uses the dedicated AI acceleration hardware integrated in the edge model execution platform to complete the model inference calculation; if the model is not deployed locally, it automatically downloads and loads a compatible version through a preset model repository, and then uses the dedicated AI acceleration hardware integrated in the edge model execution platform to complete the model inference calculation.

[0041] The model identifier uses a globally unique model fingerprint, generated jointly by the model structure hash, weight parameter hash, and version number. The model structure hash is obtained by performing a depth-first traversal of the computation graph's topology and calculating the SHA-256 digest. The weight parameter hash is calculated by sorting all floating-point weights and then calculating their Merkle root hash. The version number follows a semantic versioning specification (e.g., 1.2.3). These three components are concatenated and hashed again to form the final 64-character hexadecimal fingerprint string.

[0042] The edge-side model execution platform can pre-generate a local model cache index. This index uses the model fingerprint as the key and stores the model file path, metadata (such as input / output specifications and supported hardware backends), and performance benchmarks. Upon receiving the unload task package, the edge-side model execution platform first queries the local model cache index. If a match is found, the corresponding model is loaded directly. If not, an HTTP / 2 request is sent to the pre-defined model repository, carrying the model fingerprint as a query parameter. The model repository returns the model's metadata and the differential update package URL. The download process uses a differential update algorithm, transmitting only the changed parts. Specifically, the difference between the existing base model version on the edge-side model execution platform and the requested version is calculated using the bsdiff algorithm, and the model repository only provides this difference patch. After downloading the patch, the edge-side model execution platform uses the bspatch tool to synthesize a complete model file locally, saving over 70% of bandwidth consumption.

[0043] After the model is loaded, the edge model execution platform automatically selects the optimal hardware backend to execute the inference task based on the model's computational graph characteristics. The dedicated AI acceleration hardware includes neural network processing units, tensor computation units, or field-programmable gate arrays (FPGAs). The edge model execution platform has a built-in hardware adaptation layer, which contains runtime drivers and optimized kernels for different hardware. For example, for an NPU supporting INT8 precision, the adaptation layer automatically quantizes the model weights and activation values ​​to 8-bit integers and calls the NPU's dedicated convolution instructions; for a GPU supporting FP16, it enables half-precision tensor cores. The edge model execution platform supports multiple precision modes such as INT8 and FP16 and can dynamically switch according to application requirements. If the `target_precision` field is specified in the unloading task package, this requirement is followed first; otherwise, the edge model execution platform selects the default precision based on historical records in the hardware capability and model compatibility database.

[0044] The edge-side model execution platform can be deployed in an independent container environment on the host machine, or run on other virtual machines, client devices, edge computing nodes, or dedicated AI computing servers within the same local area network. When the edge-side model execution platform starts up, it registers its capability set (including supported model types, hardware resources, and network bandwidth) with the model scheduling manager on the host machine. The model scheduling manager selects the optimal execution node according to a preset load balancing strategy and network latency assessment model. The load balancing strategy can adopt weighted round-robin, least connections, or dynamic weight adjustment based on response time. The network latency assessment is achieved by periodically sending ICMP Echo requests or measuring RTT using custom heartbeat packets.

[0045] Step S4: Compress and encrypt the result of the model inference calculation and send it to the model request interception and forwarding module that initiated the request;

[0046] Specifically, the results of the model inference calculation are compressed and encrypted, a task identifier and integrity verification code are added, and the result is encapsulated into a response message. This message is then returned to the model request interception and forwarding module within the virtual machine that initiated the request via the original communication path or an independent high-speed backhaul channel.

[0047] The compression of the model inference calculation results can be achieved using a lossy compression algorithm based on wavelet transform. This algorithm first performs a two-dimensional discrete wavelet transform (DWT) on the output tensor, decomposing it into low-frequency approximation coefficients and high-frequency detail coefficients. Subsequently, threshold quantization is applied to the high-frequency coefficients, discarding coefficients with absolute values ​​less than the threshold and retaining only key feature information. After the inverse transform, the result volume is compressed to less than 30% of the original size, while ensuring that the key features of the model output (such as classification confidence and target detection box coordinates) are not distorted. The mean square error (MSE) between the compressed data and the original data is controlled within a preset threshold (e.g., 0.01) to ensure that downstream application logic is not affected.

[0048] The encryption process can use the national standard SM4 algorithm with a key length of 128 bits. The session key is generated by negotiation between the two communicating parties during the channel establishment phase and is extended to a subkey for result encryption through a key derivation function (KDF). The encryption operation is performed in a hardware security module (HSM) or a trusted execution environment (TEE) to prevent key leakage.

[0049] The integrity checksum can be generated using the HMAC-SHA256 algorithm. The checksum data includes task_id (task identifier, a globally unique UUID), timestamp (Unix timestamp, accurate to microseconds), and result_digest (SHA-256 digest of the result data). These three are concatenated, and an HMAC value is calculated using an independent HMAC key. The receiving end prevents data tampering and replay attacks by comparing the checksum. Specifically, after receiving a response message, the interception and forwarding module first verifies whether the timestamp is within a valid window (e.g., plus or minus 5 seconds), then recalculates the HMAC and compares it with the checksum in the message. If either verification fails, the message is discarded and a security event is reported to the scheduler manager.

[0050] The response message is returned via the original communication path. If the task is sent through shared memory, the result is also sent back through the same shared memory area and the virtual machine is notified via a reverse interrupt. If it is sent through a TLS tunnel, the result is returned along the same TCP connection to ensure sequential consistency.

[0051] Step S5: The model request interception and forwarding module receives the compressed and encrypted result of the model inference calculation, decrypts the result of the model inference calculation, and injects it into the application, so that the application can continue to execute subsequent logic and complete the entire AI model calling process.

[0052] Specifically, after receiving the response message, the model request interception and forwarding module verifies its integrity and source legitimacy, decrypts and restores the result of the model inference calculation, simulates the return format of the original model interface, and injects the result of the model inference calculation into the calling context of the application, so that the application can continue to execute subsequent logic without being aware of it, thus completing the entire AI model calling process.

[0053] The injection process of the model inference calculation results simulates the return interface behavior of the original model inference framework. The model request interception and forwarding module can generate a call context mapping table, recording the thread ID, return address, expected data structure type, and exception handling handle of each suspended call. After verifying and decrypting the model inference calculation results, the model request interception and forwarding module fills the restored tensor data into the application's output buffer according to the memory layout of the original framework (such as NHWC or NCHW), and then constructs a return code completely consistent with the original framework (e.g., 0 indicates success, non-zero indicates an error code) and an exception object (if the task execution fails). Finally, by modifying the register state of the suspended thread (e.g., RIP / EIP points to the return address, RAX / EAX stores the return value), and calling the operating system thread recovery primitive, the application thread resumes execution from the interruption point. This process maintains the consistency of the call stack, return code, and exception handling mechanism. The application can seamlessly receive remote execution results without modifying the code, and is compatible with Windows, Linux, and Unix-like operating system environments.

[0054] Furthermore, to improve robustness and resource efficiency, the method provided by this invention also includes several enhancement mechanisms. Specifically, a model scheduler can be deployed at the host level. This model scheduler monitors the computing power utilization, memory usage, and network status of each edge-side model execution platform in real time. Combining the priority and quality of service requirements of the AI ​​task, it dynamically allocates target nodes for model unloading, achieving global resource optimization. The model scheduler can communicate with the heartbeat service of each edge-side model execution platform via the gRPC interface, collecting metrics such as CPU utilization, GPU memory usage, and network throughput, and generating a real-time resource topology map. When a new model unloading task arrives, the model scheduler calculates the comprehensive score of each candidate node based on a multi-objective optimization algorithm (such as weighted summation) and selects the one with the highest score.

[0055] In addition, a model compatibility database can be established to record the performance, accuracy deviation, and resource consumption characteristics of different models on various model schedulers. This database is automatically populated from historical execution logs and includes fields such as: model fingerprint, execution platform ID, average latency, throughput, Top-1 accuracy difference between INT8 and FP32 precision, and peak memory usage. During scheduling, this database is referenced to select the optimal execution environment that meets the application requirements. For example, if the unloading task package requires accuracy_loss_budget to be less than 0.5%, the scheduler will exclude platforms where the accuracy loss on that model exceeds the threshold.

[0056] In virtual machine restart or migration scenarios, the model request interception and forwarding module saves the context of incomplete model tasks to persistent storage (such as a specific sector of the virtual machine disk or the host machine's tmpfs), restores the task state in the new operating environment, and initiates a result retransmission request to the edge-side model execution platform to ensure the continuity and reliability of AI services. The task context includes the original request data, task ID, target execution platform address, and timeout timer. After the virtual machine recovers, the model request interception and forwarding module scans the persistent storage, reconstructs the list of suspended tasks, and sends a RETRANSMIT_REQUEST message to the corresponding edge-side model execution platform. If the edge-side model execution platform still holds the result cache of the task, it directly retransmits it; otherwise, it returns an error, and the application decides whether to retry.

[0057] The method provided by this invention prioritizes offloading model call requests to local execution when the client device has AI computing power, utilizing the NPU or GPU built into the AIPC device to complete inference, reducing network transmission overhead and improving user experience. The remote offloading path is only enabled when the network is unreachable or the local computing power is insufficient. This strategy is implemented by a lightweight agent program on the client, which detects local AI acceleration hardware when the system starts and registers its capabilities with the host machine. When making decisions, the host machine's scheduler will prioritize the local offloading path if it finds that the client meets the task requirements and the network quality is good.

[0058] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for unloading AI models in a desktop cloud scenario, characterized in that, Applied to a virtual machine, wherein the virtual machine is equipped with a model request interception and forwarding module, the model request interception and forwarding module is communicatively connected to an external end-side model execution platform, the method includes: When an application in the virtual machine initiates a call request to the local AI model, the model request interception and forwarding module captures the call request and suspends its execution process on the virtual machine. The captured call request is encapsulated to generate an unload task package and sent to the client-side model execution platform; The edge model execution platform receives and parses the unloading task package, determines the corresponding executable model, and performs model inference calculation. The results of the model inference calculation are compressed, encrypted, and sent to the model request interception and forwarding module that initiated the request; The model request interception and forwarding module receives the compressed and encrypted result of the model inference calculation, and then decrypts the result of the model inference calculation and injects it into the application, so that the application can continue to execute subsequent logic and complete the entire AI model calling process. The model request interception and forwarding module is deployed in the kernel layer or user-space application layer of the operating system in the virtual machine. It achieves comprehensive capture of the AI ​​model loading and inference interface through dynamic link library injection, system call table hijacking or container-level API interception mechanism. The edge model execution platform can be deployed in an independent container environment on the host machine, other virtual machines, client devices, edge computing nodes, or dedicated AI computing power servers. The edge model execution platform selects the optimal execution node based on load balancing strategies and network latency assessment models. After receiving the unloading task package, the edge model execution platform parses the model identifier to determine the corresponding executable model instance, and then uses the dedicated AI acceleration hardware integrated in the edge model execution platform to complete the model inference calculation; if the model is not deployed locally, it automatically downloads and loads a compatible version through a preset model repository, and then uses the dedicated AI acceleration hardware integrated in the edge model execution platform to complete the model inference calculation.

2. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, The encapsulation includes mapping different vendors' proprietary API call formats to a common model task description language. The description language includes model hash value, input tensor dimension, data type, quantization method, and expected latency threshold.

3. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, The unloading task package is sent to the client-side model execution platform via a cross-virtual machine communication channel provided by the virtualization platform or a secure data channel between the host and the client.

4. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, The model identifier adopts a globally unique model fingerprint, which is jointly generated by the model structure hash, weight parameter hash and version number; the edge model execution platform matches the deployed model through the local model cache index. If no match is found, it triggers the download from the preset model repository. The download process uses a differential update algorithm to transmit only the changed parts.

5. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, The dedicated AI acceleration hardware includes a neural network processing unit, a tensor computation unit, or a field-programmable gate array. The edge model execution platform automatically selects the optimal hardware backend to execute inference tasks based on the computation graph characteristics of the model.

6. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, The results of the model inference calculation are compressed and encrypted, and a task identifier and integrity check code are attached. The result is then encapsulated into a response message and returned to the model request interception and forwarding module that initiated the request via the original communication path or an independent high-speed backhaul channel.

7. The method for unloading AI models in a desktop cloud scenario according to claim 1, characterized in that, It also includes deploying a model scheduling manager at the host level. The model scheduling manager monitors the computing power utilization, memory usage, and network status of each edge model execution platform in real time. It dynamically allocates model unloading target nodes based on the priority and service quality requirements of AI tasks. At the same time, it establishes a model compatibility database to record the performance, accuracy deviation, and resource consumption characteristics of different models on various execution platforms. During scheduling, it refers to the model compatibility database to select the optimal execution environment that meets the application requirements.

Images