Browser engine built-in gpu task scheduling method, system and computer readable storage medium

Abstract

This application discloses a GPU task scheduling method, system, and computer-readable storage medium built into a browser engine. The method includes: constructing a command queue within the browser engine corresponding to each GPU task, including rendering tasks and AI inference tasks; determining a frame time scheduling strategy based on a frame time budget and the priority of the command queue; and processing the corresponding requests according to the frame time scheduling strategy when rendering task requests and / or AI inference task requests are received from various browser tabs, and returning the processing results to the corresponding tab that initiated the request. This application balances rendering smoothness and inference efficiency. Through a frame budget model and priority rules, rendering tasks are executed first, ensuring the basic frame rate of the page and solving the problem of rendering stuttering.

Description

Technical Field

[0001] This application relates to artificial intelligence inference scheduling and browser GPU resource management technologies, and more particularly to GPU task scheduling methods, systems, and computer-readable storage media built into browser engines. Background Technology

[0002] In the field of combining artificial intelligence (AI) inference with browser applications, the browser, as the core carrier for users to access web services, urgently needs to support local AI inference capabilities to improve the interactive experience. In existing technologies, Chrome GeminiNano, as a browser-built-in AI model solution, uses a singleton ServiceController to manage the model. However, this solution does not achieve coordinated scheduling of inference and rendering tasks. When AI inference consumes GPU resources, it easily leads to browser rendering stuttering. Furthermore, when the local model is unavailable, the AI ​​API will directly fail to call, lacking a transparent local / cloud routing mechanism, requiring developers to implement fallback logic themselves. In addition, while technologies such as WebNN and ONNX Runtime Web support GPU execution of AI inference, they all load models independently at the page level, resulting in duplicate GPU memory usage. For example, a model with 7 billion parameters (7B) occupies approximately 4GB of GPU memory. Loading the same model across multiple tabs causes significant GPU memory waste. Existing technologies lack a browser-level cross-tab model sharing mechanism, failing to solve the core problems of GPU resource contention, duplicate model loading, and the separation of local / cloud inference at the system level. Summary of the Invention

[0003] This application provides a GPU task scheduling method, system, and computer storage medium built into a browser engine to solve the problem of browser rendering lag caused by AI inference consuming GPU resources.

[0004] On the one hand, the GPU task scheduling method built into the browser engine provided in this application embodiment includes: Based on the GPU task type, a command queue corresponding to each GPU task is built within the browser engine. The GPU task types include rendering tasks and AI inference tasks. The frame time scheduling strategy is determined based on the frame time budget and the command queue priority. When rendering task requests and / or AI inference task requests are received from various browser tabs, the corresponding requests are processed according to the frame time scheduling strategy, and the processing results are returned to the corresponding tab that initiated the request.

[0005] Preferably, the frame time scheduling strategy includes: Within the frame time budget, prioritize executing the command queue containing the rendering task; After the command queue containing the rendering task has been completed, determine the remaining frame time, and execute the command queue containing the AI ​​inference task during the remaining frame time and the idle time between two frames.

[0006] Preferably, the frame time scheduling strategy further includes: AI inference task requests for currently viewed tabs have higher priority than AI inference task requests for unviewed tabs.

[0007] Preferably, the method further includes: real-time monitoring of the browser engine's built-in GPU utilization, and dynamically adjusting the frame time scheduling strategy based on GPU utilization, specifically including: When GPU utilization is below a first threshold, rendering task requests and AI inference task requests are allowed to be executed in parallel; when GPU utilization is above a second threshold, rendering tasks are executed first, and AI inference task requests are executed during frame idle time; when GPU utilization is between the first and second thresholds, the frame time scheduling strategy is maintained.

[0008] Preferably, the method further includes: splitting the command queue buffer of the inference task into one or more basic inference command data blocks; When the execution time of a basic inference command data block in the AI ​​inference task is about to exceed the allocated time limit, the execution of the basic inference command data block is interrupted; the unexecuted basic inference command data block in the AI ​​inference task is pushed to the command queue corresponding to the inference task in the next frame for execution; when the allocated time limit is not exceeded, the execution of the basic inference command data block continues.

[0009] Preferably, the method further includes: Load and pool the AI ​​inference model; When an AI inference task is received, the inference state is isolated through the context cache assigned to the corresponding tab. The context cache is located within the software fault isolation security domain. Implement a least recently used model eviction policy to evict the least recently used model in the model singleton pool and retain the model currently being used by the active tab.

[0010] Preferably, the method further includes: Parse the browser's web application declaration file configuration and assess the reasoning capabilities of the device on which the browser is located; A decision matrix is ​​generated based on the inference preferences, privacy levels, cloud API addresses, and device inference capabilities configured in the declaration file. The AI ​​inference task is executed by selecting an inference path based on the decision matrix.

[0011] On the other hand, this application embodiment also provides a GPU task scheduling system built into a browser engine. This system is deployed within the browser engine and includes: the command queue module and the GPU scheduling module, wherein: The command queue module is used to build command queues corresponding to each GPU task within the browser engine according to the GPU task type, wherein the GPU task type includes rendering tasks and AI inference tasks. The GPU scheduling module is used to determine the frame time scheduling strategy based on the frame time budget and the command queue priority; when it receives rendering task requests and / or AI inference task requests from various browser tabs, it processes the corresponding requests according to the frame time scheduling strategy and returns the processing results to the corresponding tab that initiated the request.

[0012] Preferably, the frame time scheduling strategy includes: prioritizing the execution of the command queue containing the rendering task within the frame time budget; determining the remaining frame time after the command queue containing the rendering task has been completed, and executing the command queue containing the AI ​​inference task within the remaining frame time and during the idle time between two frames.

[0013] In another aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned method.

[0014] Compared with the prior art, the embodiments of this application take into account both rendering smoothness and inference efficiency. That is, through the frame budget model and priority rules, the rendering task is always executed first, ensuring the basic frame rate of the page and solving the rendering stuttering problem in the prior art. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the GPU task scheduling method built into the browser engine in an embodiment of this application; Figure 2 This is an optimization flowchart for solving the problem of repeated model loading in the embodiments of this application; Figure 3 This is an optimized flowchart for implementing transparent inference routing in an embodiment of this application; Figure 4 This is a diagram illustrating the architecture of the browser engine's built-in GPU task scheduling system in an embodiment of this application. Detailed Implementation

[0016] As mentioned earlier, in the field of integrated applications of artificial intelligence (AI) inference and browser GPU scheduling, the existing technology system includes several mainstream browser-level, operating system-level, and general-purpose AI inference-related technologies. The following section will further introduce some of these existing technologies in conjunction with the background explanation. Currently, the most commonly used solution is the Chrome Gemini Nano, a browser AI model built-in mode launched in 2024. It relies on a singleton ServiceController for model management, but only focuses on basic AI inference capability integration, without involving the coordination and scheduling of GPU inference and rendering tasks. Furthermore, when local inference capabilities are unavailable, the corresponding API will directly fail to call, lacking a transparent routing mechanism between local and cloud environments. Several common approaches exist in existing technologies: WebNN (W3C), a low-level computation graph API launched in 2023, supports GPU-based AI inference but lacks a model-level sharing mechanism, cross-browser tab scheduling capabilities, and routing functionality for local and cloud-based inference; Windows Copilot Runtime (2024), an operating system-level AI API, primarily serves system-level AI interaction and is not designed for browser scenarios, thus failing to coordinate GPU inference and rendering within the browser; DirectML, as an ML acceleration layer on DirectX 12, allows inference and rendering tasks to be interleaved, but this process requires manual management by developers and cannot achieve browser-level automatic scheduling; Android NNAPI, an operating system-level ML API, can distribute inference tasks across CPU / GPU / NPU, but uses a per-app model loading approach, lacking cross-application model sharing capabilities; Apple Core ML supports on-device ML inference and Metal GPU acceleration but does not expose related AI APIs to browsers, and Safari does not have a publicly available AI inference interface; ONNX Runtime... Although Web, as a client-side ML framework, supports WebAssembly / WebGPU backends, each page / tab loads the model independently, and there is no cross-tab model sharing mechanism.

[0017] The main drawback of the existing technologies described above is the lack of a unified GPU scheduler at the browser level to coordinate AI inference and rendering tasks. Specifically, current mainstream browsers like Chrome rely on the operating system-level GPU time-slice allocation mechanism. AI inference tasks and rendering tasks (including layax compositing, CSS animations, WebGL rendering, etc.) directly compete for GPU resources, without priority division or fine-grained time-slice allocation strategies. For example, when the browser simultaneously performs high-load AI inference and complex page rendering, the two will compete for GPU resources, causing page rendering stutters (such as dropped frames in CSS animations, WebGL screen delays), or significant delays in the AI ​​inference process, making it impossible to balance UI smoothness and inference efficiency. While DirectML can achieve interleaved inference and rendering, developers need to manually write command list management logic, which has a high technical threshold and cannot adapt to browser multi-tab and multi-application scenarios, lacking universality.

[0018] To address the GPU resource contention issue between AI inference and rendering tasks within a browser, this application provides a GPU task scheduling method built into the browser engine. See also... Figure 1 The figure shows a flowchart of this embodiment, and a detailed description is provided below with reference to the figure. In this embodiment, the specific steps are as follows: Step S11: Construct a command queue corresponding to each GPU task within the browser engine based on the GPU task type, where the GPU task types include rendering tasks and inference tasks.

[0019] This step first categorizes the tasks performed by the GPU. Generally, these can be divided into two core tasks: the first is rendering tasks, encompassing all tasks that ensure the visual presentation of the page, such as layax compositing, CSS animation execution, and WebGL rendering; the second is inference tasks, namely the forward pass computation process of the AI ​​model. Based on this task classification, the browser engine directly operates the virtual I / O GPU (virtual graphics processing unit) to access the graphics processor hardware, bypassing the operating system driver intermediary. Two independent GPU command queues are constructed within the browser engine: one is the rendering command queue (Queue 0): a high-priority queue used to store and submit command buffers corresponding to all rendering tasks, including vertex submission, texture upload, compositing blit, and image rendering instructions; the other is the inference command queue (Queue 1): a low-priority queue used to store and submit command buffers corresponding to all AI inference tasks, including model weight loading, matrix multiplication, activation function calculation, and KV cache update instructions. Furthermore, embodiments of this application can establish identifiers for the two types of tasks during GPU scheduling to ensure that the source and type of tasks are traceable, providing a basis for subsequent scheduling.

[0020] In traditional browsers, when a tab appears, the browser invokes the operating system's GPU driver to utilize the GPU. In the steps described in this embodiment, bypassing the operating system driver intermediary and directly manipulating the virtio-gpu avoids the latency and uncertainty of OS-level scheduling, allowing the browser to precisely control the execution timing of GPU tasks. This corresponds to the browser engine's built-in functionality; bypassing the operating system driver intermediary indicates that it does not rely on additional operating system components, third-party libraries, or external plugins, integrating functional modules directly into the browser engine's core program code as a native component. Furthermore, the dual-queue design in this embodiment enables logical isolation between rendering and inference tasks, laying the foundation for subsequent priority scheduling. For example, when a user simultaneously opens a tab containing AI chat functionality (triggering an inference task) and a tab with 3D animation (triggering a rendering task) in the browser, the two types of tasks will enter their respective command queues, preventing scheduling chaos caused by mixed instructions.

[0021] Step S12: Determine the frame time scheduling strategy based on the frame time budget and command queue priority.

[0022] In this step, the browser engine determines the time budget for each frame based on the screen refresh rate. For example, for a 60fps display device, the time budget per frame is 16.7ms; for a high refresh rate device of 120fps, the time budget per frame is 8.3ms. Based on this time budget, the GPU execution process for each frame is divided into three stages: Phase 1: Rendering Compositing Phase: This phase prioritizes tasks in the rendering command queue to ensure page rendering is completed within the frame budget, guaranteeing a basic frame rate. For example, in a scenario with 16.7ms / frame, the browser first schedules the rendering queue to complete the layax compositing and CSS animation rendering for the current frame. This phase is mandatory and prioritized to ensure smooth UI rendering. Phase 2: Intra-frame Inference Phase. After the rendering task is completed, the remaining time of the current frame is calculated (frame budget - rendering time). If the remaining time is greater than 1ms, an inference task chunk with the corresponding time granularity (approximately 1ms / chunk) is extracted from the inference command queue and submitted for execution. For example, if a frame takes 10ms to render and has 6.7ms remaining, six 1ms inference chunks are submitted to ensure that the inference task is executed using the intra-frame idle time and does not occupy the core rendering time. Phase 3: Inter-frame Inference Phase. After waiting for the Vertical Synchronization (VSync) signal, the idle time between two frames is entirely used to execute the remaining tasks in the inference command queue. For example, when the device is under low load, the inter-frame idle time is longer, allowing inference chunks to be executed in batches, thus improving overall inference efficiency.

[0023] In addition, this embodiment can also set priority rules: the inference request of the foreground tab (e.g., the tab currently viewed by the user) has a higher priority than the background tab (e.g., the tab that the user has not viewed), so as to ensure that the inference response of the tab currently operated by the user is faster; all rendering tasks have a higher priority than inference tasks (the core is to ensure the smoothness of the UI).

[0024] The core of the two-step strategy described above is "rendering priority, inference fill." Through phased scheduling, it ensures both page frame rate and maximizes the use of GPU idle time for inference. For example, when a user browses an AI Q&A page with dynamic charts, chart rendering (phase 1) is completed first to ensure smooth chart animation, while the inference calculation for the AI ​​answer is performed using the remaining time within the frame and the idle time between frames, avoiding inference blocking rendering.

[0025] Step S13: Dynamically adjust the frame time scheduling strategy based on GPU utilization.

[0026] In this step, the browser engine monitors GPU utilization in real time. Specifically, it reads the GPU hardware timer (gpu_timer) through the GPU scheduling module to calculate GPU utilization (i.e., the ratio of actual GPU working time to total time) in real time. The monitoring period is 1ms to ensure data real-time performance. For example, if the GPU actually works for 0.8ms within 1ms, the utilization rate is 80%. Therefore, the execution method of the inference task can be dynamically adjusted based on the utilization rate. One feasible approach is as follows: (1) When the GPU utilization is below 70%, it is determined to be a low load state, allowing inference tasks and rendering tasks to be executed in parallel, making full use of GPU resources; (2) When the GPU utilization is higher than 90%, it is judged as a high load state. The inference task is completely given up and is only executed during the inter-frame idle time to avoid system lag caused by GPU resource exhaustion. (3) When the GPU utilization is between 70% and 90%, maintain the basic frame budget scheduling strategy to balance the resource allocation for rendering and inference.

[0027] This dynamic adjustment mechanism allows for adaptation to different device load scenarios. For example, when a user opens only a lightweight AI question-answering tab (GPU utilization 60%), inference and rendering can be performed in parallel, resulting in faster AI answer generation. When a user simultaneously opens multiple high-load rendering tabs (such as 4K video playback + 3D games) and AI inference tabs (GPU utilization 95%), inference tasks are executed only between frames, prioritizing the smoothness of video and game visuals.

[0028] Step S14: Receive rendering task requests and / or AI inference task requests from each browser tab; In this step, each tab submits an inference request via the unified `ai.chat()` or `ai.embed()` API. The request includes information such as the model type (e.g., 7B) and input content (e.g., prompt text). After receiving the request, the routing decision module temporarily stores the request context to provide a basis for subsequent decisions. Of course, rendering task requests may also be received. To highlight the key points, this section mainly describes the processing of AI inference tasks.

[0029] Step S15: Process the corresponding request according to the frame time scheduling strategy and return the processing result to the corresponding tab that initiated the request; In this step, after the AI ​​inference task completes its forward propagation computation within the allocated GPU time, the GPU scheduling module sends the inference result back to the model management module. The model management module then forwards the result to the tab that initiated the request, completing the processing of one inference request. Of course, if rendering task requests from various tabs are received, they are processed according to the frame-time scheduling strategy described above.

[0030] In handling inference tasks, a preemptive execution approach can be adopted. Specifically, the browser engine breaks down the inference command buffer into preemptive chunks (inference command data blocks, inference command fragments) with smaller granularity (e.g., 1ms). During execution, the GPU scheduler checks whether the inference time budget for the current frame has been exceeded. If the execution time of an inference chunk reaches the allocated time limit (e.g., the remaining time in the frame), the execution of that chunk is immediately interrupted, and the remaining inference command data blocks are pushed to the inference queue of the next frame. If the time limit is not exceeded, the next chunk continues to be executed. This preemptive design avoids long inference tasks blocking rendering. For example, an inference chunk that requires 5ms to complete can be interrupted after 3ms if the remaining time in the frame is only 3ms. The remaining 2ms of inference command data blocks will be executed in the next frame, ensuring that the rendering tasks of each frame are completed on time and preventing frame drops due to excessively long inference tasks.

[0031] The embodiments described above effectively address the problems existing in the prior art and achieve good technical results. On one hand, the embodiments of this application balance rendering smoothness and inference efficiency. Specifically, through a frame budget model and priority rules, rendering tasks are always executed first, ensuring the basic frame rate of the page (60fps / 120fps) and solving the rendering stuttering problem in the prior art. On the other hand, inference tasks utilize the remaining time within frames and the idle time between frames, maximizing the use of GPU resources and avoiding the situation where inference tasks are completely blocked by rendering. Compared to Chrome's reliance on OS-level scheduling, inference latency can be reduced by 30%-50% in actual test scenarios. Furthermore, the solutions of the embodiments of this application require no developer intervention, significantly lowering the technical threshold: the browser engine can automatically complete GPU task classification, queue management, and scheduling. Developers only need to call a unified AI interface (such as ai.chat()), without having to manually manage the interleaved logic of inference and rendering like with DirectML, greatly reducing the development cost of browser-side AI applications. On the other hand, the embodiments of this application can achieve precise GPU resource control: bypassing the OS driver intermediary to directly operate virtio-gpu, combined with dynamic GPU utilization monitoring, can accurately control the execution timing and duration of inference tasks. Compared with the prior art, GPU resource utilization is improved by 20%-40%, and system stability problems caused by GPU overload can be avoided.

[0032] The aforementioned technical solutions are proposed for GPU scheduling. However, existing technologies may also have the following derivative drawbacks: First, repeated model loading leads to wasted GPU memory. Whether it's Chrome Gemini Nano's single-origin model management or ONNX Runtime Web's per-page model loading method, the same AI model (such as a large model of 7B size, occupying approximately 4GB of GPU memory) will be repeatedly loaded in different tabs / pages, consuming a large amount of GPU memory and reducing the overall resource utilization of the device. Second, there is a disconnect between local and cloud inference. When solutions like Chrome Gemini Nano have insufficient local inference capabilities (such as insufficient GPU memory or the model not being deployed locally), API calls will fail directly. Developers need to assess the device's capabilities and write fallback logic for cloud inference, increasing development costs and failing to guarantee the continuity and consistency of inference services. To address these drawbacks, based on the aforementioned core GPU scheduling solution, this application also provides optimization solutions for the drawbacks of repeated model loading and the disconnect between local and cloud inference.

[0033] To address the issue of repeated model loading, this application embodiment can solve it by constructing a singleton shared pool of models. See also Figure 2The diagram illustrates the process steps of this optimization technique (solving the problem of repeated model loading). Specifically, it includes: Step S21: Load and pool the AI ​​inference model. When a browser tab calls the ai.chat(“7b”, prompt) interface for the first time, the browser engine's model management module checks the built-in model singleton pool: if the “7B” model is not loaded in the model singleton pool, the model is loaded into GPU memory and registered in the model singleton pool; if the model singleton pool has already loaded the model, an inference session is directly allocated to that tab. When other tabs call the inference interface of the same model, the model weights already loaded in the model singleton pool are reused, without the need for repeated loading. Taking a 7B-scale AI model as an example, in the existing technology, each tab needs to occupy about 4GB of GPU memory to load once. If 5 tabs use the model at the same time, it needs to occupy 20GB of GPU memory; however, through model singleton pool management, it only needs to occupy 4GB of GPU memory, improving GPU memory utilization by 80%.

[0034] Step S22: When an AI inference task exists, inference state isolation is achieved through the context cache allocated to the corresponding tab. The context cache is located within the software fault isolation security domain. Model weights are shared with all tabs in read-only mode in GPU memory. Simultaneously, an independent KV Cache (context cache) is allocated for each tab's inference session, and this KV Cache is located within the corresponding tab's SFI (Security Isolation Domain). Temporary intermediate tensors generated during inference are allocated only within a single request cycle and released immediately after the request is completed. This read-only shared model weight method avoids weight redundancy across multiple tabs. The independent KV Cache ensures that the inference contexts of different tabs do not interfere with each other (e.g., the chat context of Tab A will not be leaked to Tab B). SFI isolation further enhances data security and resolves the contradiction between model sharing and data isolation in existing technologies.

[0035] Step S23: Execute the Least Recently Used (LRU) model eviction policy. When GPU memory is insufficient, the model management module triggers the LRU (Least Recently Used) eviction policy: evicts the least recently used model in the model singleton pool, while retaining the model currently being used by the active tab; the evicted model is transparently reloaded into GPU memory by the browser engine when the tab is requested again, without the developer's or user's awareness. This policy can dynamically adapt to GPU memory capacity. For example, if the device's GPU memory is 8GB and two models, 7B (4GB) and 13B (8GB), are already loaded, causing insufficient memory, if model 7B has not been used for nearly 5 minutes, model 7B is evicted, releasing 4GB of memory to ensure the normal operation of model 13B; when a tab requests model 7B later, the browser automatically reloads it, and the user only perceives a brief loading delay without manual operation.

[0036] The shortcomings of local / cloud inference separation can be addressed in this application by implementing transparent inference routing decisions. See also Figure 3 The diagram illustrates the process steps of this optimization technique (achieving transparent inference route optimization). Specifically, it includes: Step S31: The browser engine's routing decision module parses the web application's declaration file configuration; In this step, the routing decision module reads the web application's manifest file (declaration file) and extracts AI-related configurations, including inference preferences (e.g., local preferred, cloud preferred), privacy levels (e.g., local device-only inference, trusted device inference, cloud-enabled inference), and cloud API addresses. For example, if an application's manifest configuration is {"ai":{"prefer":"local","privacy":"this-device"}}, it indicates that local inference is preferred and execution is only allowed on local devices.

[0037] Step S32: The browser engine's routing decision module evaluates the device's reasoning capabilities; In this step, the routing decision module evaluates the device's hardware capabilities, including GPU memory capacity (to determine if the target model can be loaded), GPU computing power (to determine if the inference latency meets the requirements), and CPU performance (as an alternative to local CPU inference). For example, if the target model is 7B (requiring 4GB of VRAM), and the device's GPU memory is only 2GB, then the device is determined to not support local GPU inference.

[0038] Step S33: The browser engine's routing decision module generates a decision matrix based on the configuration content of the declaration file and the device's reasoning capabilities; In this step, the decision matrix can include three dimensions: device capabilities (support / not support local GPU, local CPU), privacy preferences (disable / allow cloud), and GPU load (high / medium / low). Each dimension corresponds to different decision rules. For example: if the privacy preference is "local only" + the device supports local GPU + low GPU load → select local GPU inference; if the privacy preference is "local only" + the device does not support local GPU + sufficient CPU performance → select local CPU inference; if the privacy preference is "allow cloud" + the device does not support local inference → select cloud inference (encrypted transmission of requests and results).

[0039] Step S34: The browser engine's routing decision module selects the inference path based on the decision matrix to execute the inference task; In this step, the routing decision module selects the optimal inference path according to the rules of the decision matrix. It can also record decision logs, including request time, model type, device status, and the selected inference path, for subsequent optimization of decision rules. For example, if high local CPU inference latency is repeatedly detected on a certain device, cloud inference will be prioritized (if privacy allows). If cloud inference is selected in this step, the routing decision module uses end-to-end encryption (such as TLS 1.3) to transmit inference requests and results, ensuring data privacy and preventing leakage during transmission.

[0040] The optimized routing decision scheme, based on this optimization direction, improves the flexibility and privacy security of routing decisions compared to the core scheme and existing technologies. Firstly, it ensures a multi-dimensional decision matrix, combining decisions based on privacy preferences, device capabilities, GPU load, etc., which is more adaptable to the needs of different applications (e.g., prioritizing local inference for privacy-sensitive applications and cloud inference for performance-sensitive applications) compared to the single device state decision in the core scheme. Secondly, it provides privacy-tiered control; through privacy configuration in the manifest, developers can flexibly control the transmission range of inference data, meeting the privacy compliance requirements of different scenarios compared to the lack of privacy tiers in existing technologies. Thirdly, it achieves encrypted cloud transmission: encrypted transmission during cloud inference avoids the risk of data leakage, improving data security compared to the plaintext transmission method in existing technologies.

[0041] The aforementioned optimization schemes achieve superior technical results. First, they further reduce GPU memory consumption and improve resource utilization: the singleton shared pool avoids redundant model loading across multiple tabs, reducing GPU memory usage by 50%-80% compared to existing technologies (for scenarios where the same model is used across multiple tabs). Simultaneously, the LRU eviction policy further optimizes memory allocation, enhancing the device's multitasking capabilities. Second, they significantly simplify the development process and improve service continuity: the transparent routing mechanism eliminates the need for developers to worry about device capability differences and local / cloud switching logic, while the unified ai.chat() API reduces development costs. Furthermore, automatic switching to the cloud when local inference is unavailable avoids API call failures, improving the continuity and availability of inference services by 100%. Third, they effectively balance privacy and efficiency: manifest-based privacy preference configurations can meet the privacy needs of different applications (e.g., financial applications can select "local inference only"), while the encrypted cloud inference path ensures data security during transmission, resolving the balance between privacy and efficiency in existing technologies.

[0042] Reference Figure 4 As shown in the system architecture diagram, under the optimized technical solution, this application embodiment achieves a unified three-in-one design of "GPU coordinated scheduling + model singleton sharing + transparent routing". This approach first solves the core problem of GPU resource competition between AI inference and rendering in existing technologies, ensuring a balance between browser UI smoothness and inference efficiency; secondly, it solves the problem of wasted GPU memory through the model singleton sharing pool, improving GPU resource utilization; finally, it solves the problem of local / cloud inference separation through the transparent routing mechanism, simplifying the development process and ensuring service continuity. As a unified system built into the browser engine, the overall solution fills the gap in existing technologies for the lack of a browser-level integrated solution of "scheduling + sharing + routing". Compared with existing technologies, it has achieved significant improvements in resource utilization, development convenience, and service stability, and is fully compatible with browser multi-tab and multi-application scenarios, possessing universality and practicality.

[0043] Based on the foregoing method embodiments, this application also provides an embodiment of a GPU task scheduling system built into a browser engine. This system is deployed within the browser engine and specifically includes a command queue module and a GPU scheduling module. The command queue module is used to construct command queues corresponding to each GPU task within the browser engine based on the GPU task type, which includes rendering tasks and AI inference tasks. The system may also include a model management module for managing loaded models. The GPU scheduling module is used to determine a frame time scheduling strategy based on the frame time budget and the command queue priority. When receiving rendering task requests and / or AI inference task requests from various browser tabs, it processes the corresponding requests according to the frame time scheduling strategy and returns the processing results to the corresponding tab that initiated the request. This module may also include a routing decision module for determining whether to use local or cloud resources to complete the inference task, in order to find an appropriate inference task execution device. In this system embodiment, the frame time scheduling strategy is the same as before, and may include the following: within the frame time budget, prioritize the execution of the command queue containing rendering tasks; after the command queue containing rendering tasks has been completed, determine the remaining frame time, and execute the command queue containing AI inference tasks during the remaining frame time and the idle time between two frames. This system embodiment can also be optimized in various ways to achieve specific purposes; since it corresponds to the method embodiment, it will not be repeated here. Furthermore, the technical solution of this application can be formed on a computer-readable storage medium to form a computer program.

[0044] The embodiments of this application can achieve all the inventive objectives using the aforementioned software; therefore, the software portion has been primarily described. Those skilled in the art should understand that the embodiments of this application can be provided as an apparatus, system, or related computer program product. Therefore, this application can be implemented entirely in hardware, entirely in software, or in a combination of software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0049] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0050] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0051] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0052] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A GPU task scheduling method built into a browser engine, characterized in that, The method includes: Based on the GPU task type, a command queue corresponding to each GPU task is built within the browser engine. The GPU task types include rendering tasks and AI inference tasks. The frame time scheduling strategy is determined based on the frame time budget and the command queue priority. When rendering task requests and / or AI inference task requests are received from various browser tabs, the corresponding requests are processed according to the frame time scheduling strategy, and the processing results are returned to the corresponding tab that initiated the request.

2. The method according to claim 1, characterized in that, The frame time scheduling strategy includes: Within the frame time budget, prioritize executing the command queue containing the rendering task; After the command queue containing the rendering task has been completed, determine the remaining frame time, and execute the command queue containing the AI ​​inference task during the remaining frame time and the idle time between two frames.

3. The method according to claim 2, characterized in that, The frame time scheduling strategy also includes: AI inference task requests for currently viewed tabs have higher priority than AI inference task requests for unviewed tabs.

4. The method according to claim 1, characterized in that, The method further includes: Real-time monitoring of the browser engine's built-in GPU utilization, and dynamic adjustment of the frame time scheduling strategy based on GPU utilization, specifically including: When GPU utilization is below a first threshold, rendering task requests and AI inference task requests are allowed to be executed in parallel; when GPU utilization is above a second threshold, rendering tasks are executed first, and AI inference task requests are executed during frame idle time; when GPU utilization is between the first and second thresholds, the frame time scheduling strategy is maintained.

5. The method according to claim 1, characterized in that, The method further includes: The command queue buffer for AI inference tasks is split into one or more basic inference command data blocks; When the execution time of a basic inference command data block in the AI ​​inference task is about to exceed the allocated time limit, the execution of the basic inference command data block is interrupted; the unexecuted basic inference command data block in the AI ​​inference task is pushed to the command queue corresponding to the inference task in the next frame for execution; when the allocated time limit is not exceeded, the execution of the basic inference command data block continues.

6. The method according to claim 1, characterized in that, The method further includes: Load and pool the AI ​​inference model; When an AI inference task is received, the inference state is isolated through the context cache assigned to the corresponding tab. The context cache is located within the software fault isolation security domain. Implement a least recently used model eviction policy to evict the least recently used model in the model singleton pool and retain the model currently being used by the active tab.

7. The method according to claim 1, characterized in that, The method further includes: Parse the browser's web application declaration file configuration and assess the reasoning capabilities of the device on which the browser is located; A decision matrix is ​​generated based on the inference preferences, privacy levels, cloud API addresses, and device inference capabilities outlined in the statement. The AI ​​inference task is executed by selecting an inference path based on the decision matrix.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 7.

9. A GPU task scheduling system built into a browser engine, characterized in that, The system is deployed within the browser engine and includes: a command queue module and a GPU scheduling module, among which: The command queue module is used to build command queues corresponding to each GPU task within the browser engine according to the GPU task type, wherein the GPU task type includes rendering tasks and AI inference tasks. The GPU scheduling module is used to determine the frame time scheduling strategy based on the frame time budget and the command queue priority; when it receives rendering task requests and / or AI inference task requests from various browser tabs, it processes the corresponding requests according to the frame time scheduling strategy and returns the processing results to the corresponding tab that initiated the request.

10. The system according to claim 9, characterized in that, The frame time scheduling strategy includes: Within the frame time budget, prioritize executing the command queue containing the rendering task; After the command queue containing the rendering task has been completed, determine the remaining frame time, and execute the command queue containing the AI ​​inference task during the remaining frame time and the idle time between two frames.

Images