WebGPU-based large model reasoning optimization method, system and equipment

Abstract

The invention provides a WebGPU-based large model reasoning optimization method, system and device, and relates to the technical field of generative artificial intelligence. The method comprises the following steps: in response to a request of accessing a physical GPU adapter by a browser, creating equipment and acquiring hardware features; acquiring a local large model, and performing operator fusion decision according to the hardware features, the model data and the weight data to obtain an operator fusion strategy; obtaining user input information, performing lexical element processing on the user input information to obtain input lexical element vectors, and performing granularity adjustment on the input lexical element vectors according to an operator fusion strategy and an input shape to obtain output lexical element vectors; calling an API (Application Program Interface) of the WebGPU to create a GPU buffer area, and generating a calculation pipeline and binding a resource group according to the large model calculation logic processed by the operator fusion strategy; and loading the weight data and the output lexical element vector into a GPU buffer area, and executing to obtain a reasoning result. The scheme serves as a browser end scheme for large-model localization reasoning, and user privacy can be highly guaranteed.

Description

Technical Field

[0001] This application relates to the field of generative artificial intelligence technology, and more specifically, to a method, system, and device for optimizing large model inference based on WebGPU. Background Technology

[0002] With the rapid development of artificial intelligence technology, large-scale models have become a strategic battleground in global technological competition, demonstrating enormous development potential and broad application space. Traditional large-scale model inference methods rely on cloud servers, which suffer from problems such as high deployment costs, privacy risks, response delays, and inability to be used offline.

[0003] To address this, the industry has begun exploring the use of WebGPU technology to offload AI computing tasks to users' local devices, aiming to achieve end-to-end privacy protection and low-latency interaction. However, existing browser-based inference technologies based on WebGPU face significant performance bottlenecks in practical applications. First, as the context length of processed text increases, the attention calculation mechanism of the model core consumes a sharp amount of GPU memory and memory bandwidth, becoming a key performance constraint. Second, large model inference consists of a large number of fine-grained computational operators sequentially, and existing technologies typically map these operators to independent WebGPU computing kernels, resulting in extremely high kernel startup, scheduling, and synchronization overhead.

[0004] Therefore, how to overcome existing performance bottlenecks and make full use of various optimization strategies to improve the efficiency of large model inference services in resource-constrained browser environments has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this application is to provide a method, system, and device for optimizing large model inference based on WebGPU, which can fully utilize the computing power of terminal GPUs to achieve efficient and stable large model inference.

[0006] This application is implemented as follows: In a first aspect, this application provides a large model inference optimization method based on WebGPU, comprising the following steps: responding to a browser's request to access a physical GPU adapter, creating a device and acquiring hardware features, wherein the hardware features are those acquired after the browser initializes the WebGPU environment; the device provides a local computing environment for large model inference; acquiring a local large model, and performing operator fusion decisions based on the acquired hardware features, model data, and weight data of the local large model to obtain a corresponding operator fusion strategy; acquiring user input information, and using the large model to perform lexical processing on the user input information to obtain an input lexical vector, and adjusting the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector; calling the WebGPU API interface to create a GPU buffer, and generating a corresponding computing pipeline and binding a resource group according to the computing logic of the large model processed by the operator fusion strategy; calling the WebGPU queue write interface to load the acquired weight data and output lexical vector into the GPU buffer, and executing to obtain the inference result.

[0007] Secondly, this application provides a WebGPU-based large model inference optimization system, comprising a browser unit, a large model inference service unit, and a WebGPU interface unit. The browser unit is configured to: in response to a browser request to access a physical GPU adapter, create a device and acquire hardware features, wherein the hardware features are those acquired after the browser initializes the WebGPU environment; the device provides a local computing environment for large model inference. The large model inference service unit is deployed within the browser unit and is configured to: acquire a local large model, and perform operator fusion decisions based on the hardware features acquired from the WebGPU interface unit, as well as the model data and weight data of the acquired local large model, to obtain a corresponding operator fusion strategy; utilize the large model to acquire and perform lexicalization processing on the acquired user input information to obtain an input lexical vector, and adjust the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector; call the WebGPU interface unit to create a GPU buffer, and generate corresponding computing pipelines and bind resource groups according to the computational logic of the large model processed by the operator fusion strategy. The WebGPU interface unit is configured to: provide hardware features to the large model inference service unit; in response to a call from the large model inference service unit, perform operations such as creating a GPU buffer, generating a computing pipeline and binding resource groups; and schedule the computing pipeline to perform parallel inference tasks on the weight data and output lexical vectors loaded into the GPU buffer, and return the obtained inference results to the large model inference service.

[0008] Thirdly, this application provides an electronic device including a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the method as described in any one of the first aspects above.

[0009] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0010] Fifthly, this application provides a computer program product including computer program instructions that, when executed by a processor, implement the method as described in any one of the first aspects above.

[0011] Compared with the prior art, this application has at least the following advantages or beneficial effects: This application proposes a WebGPU-based optimization method for large model inference. Through a hardware-aware operator fusion strategy, it significantly reduces the number of GPU kernel startups and intermediate data read / write operations at the source, directly overcoming the core bottlenecks of high kernel scheduling overhead and high memory bandwidth consumption. Secondly, by combining the operator fusion strategy with dynamic data optimization of the input shape, it ensures a high degree of matching between the computational task and the GPU parallel architecture, significantly improving the utilization of computational resources. Finally, these optimizations enable full utilization of the terminal's GPU computing power in resource-constrained browser environments, achieving low-latency, high-throughput, and highly stable local large model inference, effectively protecting user privacy and interactive experience. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of an embodiment of a WebGPU-based large model inference optimization method according to this application; Figure 2 This is a flowchart illustrating the specific steps of granular adjustment of the input word vector in one embodiment of this application. Figure 3 This is a structural block diagram of an embodiment of a WebGPU-based large model inference optimization system according to this application; Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0014] Icons: 101, Browser Unit; 102, Large Model Inference Service Unit; 103, WebGPU Interface Unit; 201, Processor; 202, Memory; 203, Communication Interface. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0016] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0017] Application Overview Web large-model inference utilizes WebGPU technology to offload AI computing tasks from the cloud to local devices, achieving end-to-end AI interaction while avoiding user privacy leaks and achieving extremely low latency. However, in implementing this application, the inventors discovered that existing conventional solutions directly map the fine-grained inference tasks inherent in large models to independent WebGPU computing kernels. While this "one-to-one" mapping method is intuitive, it ignores the optimal operating mode of GPUs as massively parallel processors. Each fine-grained kernel needs to be started and scheduled separately, accompanied by frequent intermediate data reads and writes, causing kernel scheduling overhead to become a bottleneck, memory bandwidth to be inefficiently occupied, and the parallel computing potential of GPUs to be not fully utilized. In addition, in real-world scenarios where user input text lengths vary (i.e., the input shape changes dynamically), existing technologies lack adaptive adjustment capabilities, and fixed computing strategies lead to frequent problems such as cache invalidation and repeated compilation, further restricting performance.

[0018] To address the aforementioned technical problems, this application provides a WebGPU-based method for optimizing large model inference. By incorporating hardware features and large model architecture features into the decision-making process, it generates an "operator fusion strategy" that can merge multiple fine-grained operators into larger computational units, thereby reducing kernel startup times and intermediate data read / write overhead. Simultaneously, before inference, based on the actual text input by the user, the aforementioned operator fusion strategy is used to preprocess and optimize the data's granularity, reducing the number of cached computational pipeline entries and preventing memory overflow. Finally, the fused and optimized computational logic is compiled into an efficient computational pipeline for parallel inference execution on the GPU.

[0019] That is, this application reshapes fragmented computational task flows into coarse-grained computational task packages suitable for large-scale parallel processing of GPUs through hardware-aware operator fusion decision-making and dynamic optimization that adapts to user input data. This enables the full utilization of the computing power of the terminal GPU to achieve efficient and stable large model inference.

[0020] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0021] Exemplary methods Please see Figure 1 This WebGPU-based large model inference optimization method includes the following steps: Step S101: In response to the browser's request to access the physical GPU adapter, a device is created and hardware characteristics are obtained. The hardware characteristics are the corresponding hardware characteristics obtained after the browser initializes the WebGPU environment. The device provides a local computing environment for large model inference.

[0022] In step S101 above, after the user triggers a smart interaction request in the browser, the browser first requests and initializes the GPU device through the standard WebGPU API. During this process, it actively queries the specific performance limit parameters of the device, such as the maximum workgroup storage size, the maximum workgroup size, and device architecture information. These parameters objectively characterize the current GPU's parallel computing capabilities and memory resource limits. This transforms the abstract GPU computing power into a series of specific and quantifiable hardware constraints, ensuring that the generated optimization strategy (operator fusion strategy in step 102) and computing kernels (computing pipelines and bound resource groups in step 104) do not exceed the actual capabilities of the current device, which is the cornerstone of achieving hardware adaptive optimization.

[0023] It's worth noting that when responding to intelligent interaction requests, the system can first verify if the browser has the capability to run WebGPU. If not, it returns a message suggesting the user switch to a browser that supports WebGPU. For example, when suggesting a suitable browser, it can simultaneously provide recommendations for the corresponding browser and version. If available, the browser requests access to the physical GPU adapter, creating a device as the foundation for all subsequent GPU operations, providing a local computing environment for large model inference. To ensure a high level of protection for user privacy data, all target objects to be operated on after the WebGPU environment is initialized must be imported into a sandbox to ensure the security of the subsequent large model loading channel and isolate local data storage, fundamentally preventing any behavior that sends user privacy data to external servers.

[0024] Step S102: Obtain the local large model, and make operator fusion decisions based on the obtained hardware features, model data and weight data of the local large model to obtain the corresponding operator fusion strategy.

[0025] In step S102 above, the large model inference service, based on the model data (operator features, dependency graph) and weight data of the local large model, combined with the hardware features obtained in step S101, inputs them into an optimization decision process. This process, by analyzing the data dependencies and computational rules between operators, intelligently decides, under the current hardware constraints, which consecutive fine-grained operators (e.g., "linear layer → activation function → layer normalization") should be merged into a coarser-grained composite computational unit, and determines the optimal execution parameters of this unit (e.g., memory block size), ultimately outputting a complete operator fusion strategy.

[0026] It should be noted that, since this operator fusion strategy is an optimal fusion scheme for specific local large models and specific hardware characteristics, it reduces the total number of GPU cores that need to be launched by merging multiple small operators and eliminates a large number of intermediate result read / write operations between operators. This directly and effectively overcomes the two major pain points of "high kernel scheduling and synchronization overhead" and "large memory bandwidth consumption", laying the foundation for significantly improving computational efficiency.

[0027] It should also be noted that, in order to reduce the memory usage and inference overhead of large models in a browser environment, and to enable them to load and run efficiently, large models generally need to be quantized to compress their size (small models with sufficient hardware resources do not require compression). Meanwhile, to avoid the large model failing to leverage the advantages of quantization due to changes in operator data types after quantization, and to maximize the computational efficiency benefits of operator fusion strategies, this application first compresses the large model size through quantization, and then optimizes the computational logic of the quantized large model using operator fusion strategies. This allows quantization parameters to guide operator optimization, achieving the best effect of quantization and operator co-design, thereby enabling the large model to achieve the goals of "accuracy met, latency optimal, and memory controllable" during inference. For example, quantization can use algorithms such as GPTQ (Generative Pre-trained Transformer Quantization) to perform high-precision quantization on the large model, compressing the large model weights from 16-bit floating-point numbers to 4-bit integers without significantly sacrificing performance, thus reducing the resource consumption of the large model.

[0028] For example, before making operator fusion decisions based on the hardware characteristics and the acquired model data and weight data of the local large model to obtain the corresponding operator fusion strategy, it is possible to first determine whether the large model has been used locally before based on its name and version number. If it has been used, the previously recorded operator fusion strategy is directly reused, establishing a mapping relationship between model identifiers and fusion schemes, reducing computational redundancy and minimizing repetitive overhead. If the large model has not been used, then the step of making operator fusion decisions based on the hardware characteristics and the acquired model data and weight data of the local large model to obtain the corresponding operator fusion strategy is performed.

[0029] Furthermore, once the large model is loaded and cached upon first use, the large model inference service can run completely offline, solving the problem of large models being available offline in certain special circumstances and avoiding excessive reliance on service providers. For example, the model data of a large model includes at least operator features (such as operator type identifiers like convolution and activation functions; structural parameters like convolution kernel size and stride; and data features like input tensor shape and quantization precision) and dependency graphs (such as topological structures like the execution order and data flow between operators).

[0030] Step S103: Obtain user input information, and use the large model to perform lexicalization on the user input information to obtain input lexical vectors. Then, adjust the granularity of the input lexical vectors according to the operator fusion strategy and the input shape of the input lexical vectors to obtain output lexical vectors.

[0031] In step S103 above, the user input information is the information that the user enters into the input box on the web page to query. The large model inference service uses the user input information it obtains to perform lexical processing on the large model to obtain the input lexical vector. Lexical processing is to divide the original text into smaller units (lexicals) and convert them into numerical sequences. In order to facilitate the large model's later calculations, these lexicals can be words, subwords or characters.

[0032] Next, after the large model inference service obtains the numerical sequence (input word vector), it will preprocess the input data according to the optimization method for the data shape of different input word vectors in the operator fusion strategy generated in step 102, and in combination with the actual sequence length of this input. For example, it will group and merge excessively short sequences, or reasonably split excessively long sequences, and finally output an output word vector with a data granularity that is more matched with the parallel computing characteristics of the GPU.

[0033] Thus, by combining operator fusion strategies and adjusting the granularity of the input lexical vectors based on their shape, we can ensure that the organization of the data to be processed maximizes the parallel utilization of the GPU. By making the size of the data blocks more closely match the execution granularity of the GPU thread groups, we can significantly reduce GPU core idle time and improve computational density. Simultaneously, adaptive adjustments for inputs of different lengths enhance robustness in handling diverse inputs, avoiding drastic performance fluctuations caused by changes in input shape.

[0034] It should be noted that input shape refers to the dimension and size of the user input information, such as the sequence length and number of tokens in the text. If the inference operator granularity is at the token level, given that the context of each token may differ, this will lead to an excessive number of cached items, resulting in low cache utilization and a large query open volume. This application adjusts the granularity of the input token vector based on an operator fusion strategy to reduce the number of tokens to be inferred, thereby reducing the query open volume and the amount of objects generated in the subsequent computation pipeline, and avoiding memory overflow. Moreover, during the computation process, fusion operators can reduce the data read / write overhead between multiple consecutively executed independent operators, reducing memory bandwidth pressure.

[0035] Step S104: Call the WebGPU API interface to create a GPU buffer, and generate the corresponding computing pipeline and bind the resource group according to the computing logic of the large model after the operator fusion strategy.

[0036] In step S104 above, the large model inference service first calls the WebGPU buffer creation interface, which allocates buffer space in memory to store the weight data, input data, and output results of the large model. Next, the large model inference service converts the various layers and operations (such as matrix operations and activation functions) corresponding to the operator-fused large model into computation shader code. Then, the large model inference service calls the WebGPU shader module creation interface, which compiles this shader code to generate executable shader modules. Finally, the large model inference service calls the WebGPU pipeline creation interface to create a pipeline layout to declare the resources (such as buffers and textures) accessible to the computation shaders and their binding rules. WebGPU then combines the shader modules and pipeline layout to generate computation pipeline objects and bound resource groups.

[0037] In this way, the operator fusion strategy can be solidified into efficient, reproducible GPU machine code. The generated computation pipeline is a deeply optimized, immutable object that the GPU driver can optimize at the lowest level. Binding resource groups decouples the computational logic from the actual data, allowing the same optimized code (pipeline) to efficiently process different batches of input data. The combination of these two approaches provides a stable and efficient computational module for the final high-performance execution, and its compilation overhead can be amortized through subsequent reuse mechanisms.

[0038] The reuse mechanism refers to the ability, after creating a GPU buffer, to determine whether a computation pipeline has been generated before in the historical creation record based on the GPU buffer, the type and version information of the large model, and if so, to reuse the computation pipeline and the bound resource group, thereby significantly reducing the overhead of pipeline compilation and initialization.

[0039] Step S105: By calling the WebGPU queue write interface, the obtained weight data and output word vectors are loaded into the GPU buffer, and the inference results are obtained.

[0040] In step S105, the weight data of the large model and the optimized output lexical vectors from step S103 are first loaded into the GPU buffer. Then, by calling the WebGPU API interface based on the computation pipeline generated in step S104 and passing the corresponding bound resource groups, thousands of GPU cores are driven to work in parallel to complete the inference computation of the entire large model, ultimately obtaining the corresponding inference result. Since the user input information has been optimized (step S103), the execution code is highly optimized (step S104), and the hardware resources are perfectly adapted (steps S101 and S102), the GPU can execute tasks with the highest parallelism and computational density while minimizing kernel scheduling overhead and memory access latency. Therefore, step S105 achieves extremely low single-inference latency and extremely high throughput, making smooth, real-time interaction with the large model on the browser side possible.

[0041] In summary, the above embodiments are initiated after a user triggers an intelligent interaction request. First, the browser initializes the WebGPU environment and acquires GPU hardware characteristics, establishing a hardware-aware foundation for subsequent optimization. Then, the large model inference service acquires the local large model and inputs the hardware characteristics, model data, and weight data of the large model into the intelligent decision-making process, generating an "operator fusion strategy." This operator fusion strategy, tailored to the specific hardware and large model, guides the merging of multiple consecutive fine-grained operators into coarser-grained computational units. Based on this operator fusion strategy, the user input text is dynamically optimized: first, it is converted into a numerical vector, and then the data granularity is adjusted according to the operator fusion strategy and the input shape, forming a more regularized data block suitable for GPU parallel processing. Simultaneously, the large model computation logic is compiled into a highly optimized WebGPU computation pipeline and corresponding binding resource groups based on operator fusion. Finally, the weight data and output lexical vectors are loaded into the GPU buffer, driving the computation pipeline to execute efficiently in parallel, completing the inference.

[0042] In other words, the above embodiments, through a hardware-aware operator fusion strategy, significantly reduce the number of GPU kernel startups and intermediate data read / write operations at the source, directly overcoming the core bottlenecks of high kernel scheduling overhead and high memory bandwidth consumption. Secondly, by combining the operator fusion strategy with dynamic data optimization of the input shape, a high degree of matching between the computational task and the GPU parallel architecture is ensured, significantly improving the utilization of computing resources. Finally, these optimizations enable full utilization of the terminal's GPU computing power in resource-constrained browser environments, achieving low-latency, high-throughput, and highly stable local large-model inference, effectively protecting user privacy and interactive experience.

[0043] Based on the aforementioned scheme, in some implementations of this application, in the step of creating a device and obtaining hardware characteristics in response to a browser's request to access the physical GPU adapter, the method for obtaining hardware characteristics includes: the browser requests the GPU adapter and device to initialize the WebGPU environment; the browser queries the maximum workgroup storage size, maximum workgroup size, and device architecture information from the obtained GPU device object to use as the hardware characteristics and transmits them to the large model inference service.

[0044] In the above implementation, an accurate hardware profile is provided by transforming abstract performance concepts into specific, quantifiable, and verifiable hardware limit parameters in the WebGPU standard. Among them, the maximum workgroup storage size determines the upper limit of high-speed shared memory available to each computing unit and is the core basis for designing data partitioning strategies; the maximum workgroup size limits the thread scale of a single parallel computation and directly affects the granularity of task partitioning; and device architecture information can be used to infer its performance level, such as memory bandwidth and number of cores.

[0045] It should be noted that these specific parameters ensure that the subsequently generated operator fusion strategy and computing pipeline will not exceed the physical limits of the current device, thus avoiding resource allocation failures or performance backlash caused by blind optimization.

[0046] Based on the aforementioned scheme, in some implementations of this application, the step of performing operator fusion decision based on the acquired hardware features, model data of the local large model, and weight data to obtain the corresponding operator fusion strategy includes: taking the model data, weight data, and hardware features as input, and calling a supervised learning model to perform operator fusion decision to obtain the corresponding operator fusion strategy. The supervised learning model is a machine learning model trained based on historical model data, weight data, hardware features, and operator fusion strategy. It minimizes a preset optimization objective by analyzing hardware features and constraints on the current large model's model data and weight data, and outputs an operator fusion strategy containing at least one fusion operator definition and corresponding execution parameters. The preset optimization objective includes at least one dimension among execution time, resource consumption, hardware adaptability, reusability, or dynamic scene adaptability.

[0047] In the above implementation, a pre-trained supervised learning model is introduced as the decision engine. During inference, hardware features provide physical constraints, model data reveals the computational graph structure, and weight data reflects the computational load. Specifically, this supervised learning model uses model data from the current large model (such as operator features and dependency graphs), weight data, and specific hardware features obtained from the WebGPU as joint inputs. This supervised learning model is trained on a large amount of historical data (covering different models, hardware configurations, and validated optimal operator fusion strategies). Its training objective aims to minimize a multi-dimensional comprehensive optimization objective, which covers at least one aspect: execution time, resource consumption, hardware adaptability, reusability, or dynamic scene adaptability.

[0048] It should be noted that the above implementation method, by comprehensively considering multiple objectives such as execution time and resource consumption, achieves the best balance between reducing kernel overhead, saving memory bandwidth and improving hardware utilization, thereby fundamentally and systematically improving the execution efficiency of subsequent WebGPU computing pipelines.

[0049] For example, a supervised learning model can be a random forest model, a gradient boosting decision tree model, or a neural network model.

[0050] Based on the aforementioned scheme, in some implementations of this application, the model data includes operator features and dependency graphs; the step of performing operator fusion decision based on the acquired hardware features, local large model model data, and weight data to obtain the corresponding operator fusion strategy includes: using operator features, dependency graphs, and weight data as input, performing feature extraction using a graph neural network model to obtain the embedding vector of each operator and the dependency matrix representing the topological structure between operators; inputting the embedding vector, dependency matrix, and hardware features into a gradient boosting decision tree model, which processes the data according to preset multi-dimensional selection criteria to obtain the corresponding operator fusion strategy, wherein the multi-dimensional selection criteria include at least one of execution time optimization, resource consumption reduction, and hardware adaptability.

[0051] In the above implementation, the supervised learning model is designed as a two-stage model that combines graph neural networks with gradient boosting decision trees.

[0052] The first stage is the feature extraction stage, which uses a graph neural network (GNN) model to perform deep feature extraction on the computation graph of the large model. Operator features (including operator type identifiers such as convolution and activation functions, structural parameters such as kernel size and stride, and data features such as input tensor shape and quantization precision), operator dependency graphs (including operator execution order, data flow, and topology), and weight data are used as input information. The large model inference service asynchronously calls the API of the graph neural network model through a task queue to perform feature extraction, outputting the embedding vector of each operator and the global dependency matrix.

[0053] The second stage is the strategy decision stage. The operator embedding vector, dependency matrix, and hardware features (hardware configuration parameters, including thread parallelism, shared memory limit, memory bandwidth, etc.) output from the first stage are used as input information. The large model inference service calls the API of the gradient boosting decision tree (GBDT) model for processing. This model takes minimizing the prediction execution time as the objective function, comprehensively considers the computation time, memory access time, and synchronization overhead, and outputs the optimal fusion operator scheme.

[0054] For ease of understanding, the following example demonstrates the multi-dimensional selection criteria that a supervised learning model can follow when generating fusion solutions. It should be noted that these criteria work together in the decision-making process to achieve a comprehensive optimization objective: (1) Execution time optimization: The prediction execution time of the fused operator should be lower than that of the original strategy by a preset ratio (e.g., a reduction of 20%). Specifically, the overall computation time is optimized by eliminating intermediate result read and write to reduce memory access time, reducing the number of operators to reduce kernel synchronization overhead, and increasing computation density.

[0055] (2) Resource consumption: Reduce video memory usage by eliminating intermediate tensor storage, reduce unnecessary data transfer to improve memory bandwidth utilization, and reduce the number of generated WebGPU pipelines to reduce the overhead of resource binding and management.

[0056] (3) Hardware compatibility: Ensure that the number of workgroups specified in the generation scheme can fully utilize the parallel computing capabilities of the GPU, the demand for shared memory does not exceed the actual upper limit of the hardware, and the thread organization method should be aligned with the GPU's Warp / Wavefront size to match the underlying hardware execution unit.

[0057] (4) Reusability: When making decisions, priority is given to reusing existing historical fusion schemes and coordinating with the shape bucketing mechanism (see below for details) so that different inputs within the same bucket can share the same fusion configuration and cache resources.

[0058] (5) Dynamic Adaptability Support: Ensure that the generated fusion scheme can adapt to different input shapes and batch sizes, and has a certain degree of cross-model universality and scalability to cope with diverse application scenarios.

[0059] Based on the aforementioned solutions, such as Figure 2 As shown, in some implementations of this application, the step of adjusting the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain the output lexical vector includes: Step S201: Mapping the input lexical vector to a predefined shape bucket according to the sequence length of the input lexical vector; Step S202: Obtaining the fusion operator configuration instantiated by the operator fusion strategy corresponding to the shape bucket; Step S203: Grouping and / or splitting the input lexical vector based on the block parameters in the fusion operator configuration to obtain the corresponding output lexical vector.

[0060] In the above implementation, to address the fragmentation problem caused by the diversity of input word vector shapes, a shape bucketing mechanism is introduced to achieve dynamic optimization when adjusting the granularity of the input word vector. After obtaining the input word vector, it is first categorized into several predefined shape buckets (such as short sequence buckets, medium-to-long sequence buckets, etc.) based on its sequence length. Each shape bucket is associated with a specific fusion operator configuration instantiated by the aforementioned operator fusion strategy. This configuration contains block parameters optimized for the data shape range within that bucket. Finally, based on these block parameters, grouping (merging small data blocks) or splitting (splitting large data blocks) operations are performed on the input word vector to generate a more granular and regular output word vector.

[0061] Thus, the above implementation transforms the dynamically changing input shape problem into a limited number of optimization scenarios through shape binning, and combines this with fusion operator configuration to achieve collaborative optimization of data preprocessing and computation strategies. It avoids the enormous overhead of recompiling the computation kernel for every possible input shape and reduces cache fragmentation caused by shape differences. Inputs within the same bin can reuse the same highly optimized fusion operator configuration and cache pool, significantly reducing query and compilation overhead and improving cache utilization. Ultimately, it ensures that regardless of the length of user input, it provides "ready" high-quality input for subsequent GPU parallel computing in a near-optimal data organization manner, guaranteeing stable and efficient inference performance.

[0062] For example, the above shape bucketing mechanism may include: when an input token vector arrives, determining its bucket based on its sequence length, and mapping the input to the corresponding bucket's cache pool and operator configuration. If the sequence length does not perfectly match the standard shape of the bucket, a padding strategy is used for alignment.

[0063] The shape of the input lexical vector is (B, L, D), where: B represents the batch of user input information; L represents the sequence length and the number of lexical units; and D represents the hidden layer dimension.

[0064] In the inference computation function, the bucket to be placed is first determined based on the sequence length of the input tokens. Then, the input tokens are divided into blocks according to the pre-configured Tiling parameters of that bucket. The Tiling parameters define the size of the data block processed in each computation, and are used to optimize memory access patterns and computational parallelism. It should be noted that the Tiling parameters refer to a set of quantization rules in GPU computing that define how to systematically divide large-scale multidimensional data (such as matrices, tensors) or computational domains into smaller, regularly shaped, and fixed-size data blocks (called tiles or blocks).

[0065] For example, the buckets it is divided into may include: Short sequence buckets: Sequence length 1-32 words, corresponding to shapes such as (1, 25, 768); Medium-length text buckets: sequence length 33-64 words, corresponding to shapes such as (1, 50, 768). Long text bucket: The sequence length is 65-128 words, and the corresponding shape is (1, 100, 768).

[0066] Based on the aforementioned scheme, in some implementations of this application, the step of generating a corresponding computing pipeline and binding a resource group according to the computing logic of the large model processed by the operator fusion strategy includes: determining whether there is a historical computing pipeline record based on the GPU buffer and the identification information of the large model. If it exists, the computing pipeline and resource group in the historical record are reused; if it does not exist, the computing logic of the large model processed by the operator fusion strategy is compiled into a WebGPU computing shader, and a new computing pipeline is created and a resource group is bound based on the computing shader.

[0067] Since compiling and optimizing the WebGPU computing pipeline is a relatively time-consuming process, the above implementation utilizes a reuse mechanism. This allows subsequent inference requests for the same large model on the same device to completely skip the time-consuming compilation phase and directly use the deeply optimized pipeline objects, thus minimizing inference startup time. This not only improves the response speed of single interactions but also significantly enhances throughput and latency consistency in continuous interaction scenarios. Simultaneously, this reuse mechanism reduces redundant consumption of browser and GPU driver resources, resulting in more efficient overall resource utilization and providing users with a smoother and more stable local AI interaction experience.

[0068] Based on the aforementioned scheme, in some implementations of this application, the step of loading the weight data and output lexical vector into the GPU buffer to perform inference tasks includes: loading the weight data and output lexical vector into the GPU buffer, and recording GPU instructions through a command encoder to generate a command buffer; submitting the command buffer to the GPU command queue to schedule the computing pipeline to execute the inference task in parallel.

[0069] In the above implementation, after loading the weight data and output lexical vectors of the large model into the GPU buffer, the WebGPU command encoder records the operations that the GPU needs to perform (such as setting up the computing pipeline, binding resource groups, scheduling the number of workgroups, etc.) as a series of low-level instructions and encapsulates them into a command buffer. Then, this command buffer is submitted to the GPU's command queue, where the GPU driver asynchronously retrieves and executes the commands, thereby scheduling the created computing pipeline and driving the GPU's numerous computing cores to complete the entire inference task in parallel.

[0070] For example, the large model inference service obtains the weight data and output word vectors of the large model by calling the WebGPU queue write interface and loads them into the GPU buffer. Then, it performs block processing on the inference task, creates a command encoder to record GPU instructions, and submits the generated command buffer to the GPU queue for execution. The steps may include: First, the large model inference service binds GPU resource groups (including buffers for storing model weights, buffers for storing output word vectors, and buffers for storing computation results) to resource slots defined in the computation pipeline based on binding rules, establishing a mapping relationship between the computation pipeline and data resources. Then, the large model inference service loads the large model's weight data and output word vectors into the corresponding GPU buffers via the WebGPU API. The large model inference service then processes the complex inference task formed after updating the model weight data and output word vectors in blocks. Finally, based on all the inference tasks formed after separate processing, WebGPU generates a command encoder, records GPU computation instructions through the command encoder, generates a command buffer after encoding, and submits the command buffer to the GPU queue for execution.

[0071] Based on the aforementioned scheme, in some implementations of this application, after the step of loading the acquired weight data and output word vectors into the GPU buffer, the complex inference task formed after updating the model weight data and output word vectors is divided into blocks. The specific steps include: first, loading the weight data and output word vectors into the GPU buffer to obtain a queue of tasks to be inferred. Each task to be inferred in the queue is a computation task that contains each corresponding output word vector and calculation rule and is to be executed in parallel by the GPU.

[0072] Subsequently, based on the weight data, each output lexical vector, and the hardware features, the computational task complexity is quantified to obtain performance metrics. The calculation method for performance metrics is as follows:

[0073] For performance indicators; The total number of floating-point operations for the inference task is obtained based on the shape of each different output lexical vector, the dimension of the weight data, and the computational complexity of the fusion operator. Maximum size for the workgroup; This is the GPU warp size (typically 32 for NVIDIA, 64 for AMD). Maximum workgroup storage size; Memory usage is allocated for a single block of data.

[0074] Finally, based on the performance metrics corresponding to each inference task, it is determined whether a preset threshold is exceeded. If so, the inference task is decomposed into multiple fine-grained computing units executed in parallel by GPU thread groups based on a block-based strategy. If the computational load exceeds a preset threshold, it indicates that the task's computational load exceeds the GPU's capacity for single-pass parallel processing, and is thus classified as a "complex task," requiring block processing based on a block-based strategy; if If the value is less than or equal to the preset threshold, it means that the task can be completed in parallel execution by a single GPU without the need for block division.

[0075] It should be noted that the above classification strategy aims to decompose complex computational tasks into fine-grained units suitable for GPU parallel architecture, and to efficiently utilize GPU memory and computing cores. Before the WebGPU command encoder records GPU instructions, the fusion operators of large models are decomposed into multiple smaller operators. Data reuse is achieved by utilizing GPU shared memory. Through task parallelization and resource optimization, GPU utilization is significantly improved and memory access overhead is reduced. The specific decomposition method is as follows: For example: large matrix and Decompose into smaller chunks and utilize GPU shared memory to accelerate computation. Mathematical representation: Let matrix ,in:

[0076]

[0077]

[0078] in, This is the first input matrix; This is the second input matrix; This is the output matrix; for the number of rows, number of rows; for The number of columns, number of rows; for The number of columns, The number of columns.

[0079] Divide the matrix into blocks: matrix Decomposed into Sub-blocks, matrix Decomposed into Individual blocks.

[0080] in, for The number of blocks divided in the row direction, taking the value of ; for In column direction ( The number of blocks (in the row direction) that is divided into, taking values ​​of ; for The number of blocks divided in the column direction, taking the value of ; For each block The size of the dimension (row direction); For each block The size of the dimension (intermediate dimension); For each block The size of the dimension (column direction).

[0081]

[0082] Then each block in the result matrix : ,in, ,

[0083] Each sub-block , , .

[0084] in, for The line block, first Column blocks, in which , ; for The line block, first Column blocks, in which , ; for The line block, first Column blocks, in which , .

[0085] Block size selection:

[0086] in, For the final selected block, To obtain the minimum value, The total size of GPU shared memory (in bytes); The number of matrix blocks that need to be cached simultaneously ( , ); The size in bytes of the data type (e.g., float32 is 4 bytes); Maximum size limit for GPU working groups; The tuning factor (usually ranging from 100 to 1000) ); The GPU warp size (typically 32 for NVIDIA, 64 for AMD) express).

[0087] In summary, the WebGPU-based large model inference optimization method provided in the "Exemplary Method" section can fully utilize the computing power of the terminal GPU to achieve efficient and stable large model inference. If implemented as a system, it constructs a closed-loop system of "front-end interaction - task inference - hardware acceleration" through the collaboration of the browser, inference module, and WebGPU, forming a new paradigm for browser-side AI inference. The structure of this paradigm can be found in the "Exemplary System" section below and will not be elaborated further here.

[0088] It should be noted that, to facilitate a more intuitive understanding of the technical advantages of this application by those skilled in the art, the following will provide exemplary demonstrations in conjunction with the overall technical paradigm of this application and some optional implementation details. It should be specifically noted that the following demonstrations are intended to aid understanding and do not constitute an exhaustive list of all embodiments of this application, nor do they imply that this application must include all the details described below in its specific implementation.

[0089] In this closed-loop system formed by the collaboration of the browser, the large model inference service, and WebGPU, which consists of "front-end interaction - task inference - hardware acceleration", the browser, as the runtime environment layer, is responsible for providing the native isolated computing and storage environment required to run the large model inference service. This ensures that data and computing are completed locally, thereby protecting user privacy and security from the source and avoiding network latency.

[0090] The large-scale model inference service, deployed in the browser as the business layer, is the core of intelligent processing. While utilizing user input information from the web application, it also comprehensively considers factors such as the operator features of the large model, dependencies between operators, and hardware configuration parameters obtained from the WebGPU. It calls the supervised learning model API to perform operator fusion decisions on the computational logic of the large model, generating an operator fusion strategy. This aims to improve computational efficiency, reduce resource consumption, and enhance hardware adaptability. Subsequently, the user input information obtained from the web application by the large model is used for lexicalization to obtain input lexical vectors. For further optimization, the large model adjusts the granularity of the input lexical vectors based on the operator fusion strategy and the shape of the input lexical vectors (e.g., merging small-granularity lexical elements) to obtain the output lexical vectors. This process reduces the number of pending lexical cache items and the number of operators in the original computation graph. That is, it can reduce the number of speculative meta-caches and operators of large model inference computation logic, thereby reducing the number of workgroups for subsequent WebGPU computing tasks, and also reducing the number of subsequent computation pipeline objects created, reducing memory pressure. In particular, as the context length increases, it can avoid memory overflow caused by too many cached computation pipeline objects.

[0091] Furthermore, to address the cache fragmentation issue caused by the diverse shapes of input lexical vectors, a shape-based bucketing mechanism is introduced to bucket the input lexical vectors (e.g., by sequence length). This mechanism works in conjunction with the operator fusion mechanism, ensuring that data within the same bucket shares the same fusion operator configuration and cache pool (i.e., using the same Tiling parameters and cache configuration). This effectively avoids duplicate compilation and cache invalidation caused by shape differences, further reducing the number of cached items and query overhead.

[0092] WebGPU, acting as a hardware interface acceleration layer, provides hardware features to the large model inference service and responds to its calls to offer parallel computing capabilities for tensor operations and efficient memory management. The large model inference service calls the WebGPU API, which compiles the computational logic of the large model after operator fusion into a computational pipeline. The inference task is then divided into blocks, breaking it down into fine-grained computational units (i.e., small operators) suitable for parallel GPU execution. The large model inference service loads model weight data and output word vectors into a GPU buffer created by WebGPU, records GPU instructions through a command encoder to generate a command buffer, submits it to the GPU command queue, and then the GPU performs parallel inference computations and returns the results.

[0093] The closed-loop system formed by the browser, inference module, and WebGPU—comprising "front-end interaction - task inference - hardware acceleration"—not only reduces the basic overhead of kernel scheduling and resource binding but also improves edge-side inference throughput and latency consistency. Furthermore, operator reuse and computation pipeline reuse mechanisms further reduce redundant compilation and resource binding overhead, improve parallel efficiency, and significantly enhance the performance and resource utilization of GPU applications.

[0094] In addition, for large model inference tasks, the processing paradigm of first fusing (reducing the number of operators and pipelines) and then splitting (decomposing the task into GPU-friendly units) significantly improves GPU utilization and reduces memory access overhead, enabling WebGPU to achieve efficient parallel computing and thus accelerate large model inference.

[0095] Furthermore, based on the browser's native isolation and access control mechanisms, this application ensures that all data transmission and computation involved in AI inference are completed locally, avoiding the uploading of sensitive personal information to the server and thus reducing the risk of privacy leaks. Simultaneously, local computation is not limited by network bandwidth and round-trip latency, significantly improving interaction latency and response stability, and enhancing user experience. Moreover, after the initial loading and weight caching, the large model can continue inference locally on the webpage offline, without the need for servers, internet access, or cloud computing power, while simultaneously ensuring both privacy protection and a low-latency interactive experience.

[0096] Exemplary System Please see Figure 3 This application provides a WebGPU-based large model inference optimization system, which includes a browser unit 101, a large model inference service unit 102, and a WebGPU interface unit 103.

[0097] The browser unit 101 is configured to: respond to a browser's request to access the physical GPU adapter, create a device and acquire hardware features, which are obtained after the browser initializes the WebGPU environment; the device provides a local computing environment for large model inference. The large model inference service unit 102 is deployed in the browser unit 101 and configured to: acquire a local large model, and perform operator fusion decisions based on the hardware features acquired from the WebGPU interface unit 103, as well as the acquired model data and weight data of the local large model, to obtain a corresponding operator fusion strategy; use the large model to acquire and perform lexical processing on the acquired user input information to obtain an input lexical vector, and adjust the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector; call the WebGPU interface unit 103 to create a GPU buffer, and generate corresponding computing pipelines and bind resource groups according to the computing logic of the large model processed by the operator fusion strategy. WebGPU interface unit 103 is configured to: provide hardware features to the large model inference service unit 102; in response to a call from the large model inference service unit 102, perform operations such as creating a GPU buffer, generating a computing pipeline and binding resource groups; and schedule the computing pipeline to perform parallel inference tasks on the weight data and output lexical vectors loaded into the GPU buffer, and return the obtained inference results to the large model inference service.

[0098] For details on the implementation of the above system, please refer to the WebGPU-based large model inference optimization method provided in the "Exemplary Methods" section, which will not be repeated here.

[0099] Based on the aforementioned scheme, in some implementations of this application, the step of creating a device and obtaining hardware characteristics in response to a browser's request to access the physical GPU adapter, the step of obtaining the hardware characteristics includes: the browser requests the GPU adapter and device to initialize the WebGPU environment; the browser queries the maximum workgroup storage size, maximum workgroup size and device architecture information from the obtained GPU device object as the hardware characteristics.

[0100] Based on the aforementioned scheme, in some implementations of this application, the step of making operator fusion decisions based on the acquired hardware features and the model data and weight data of the local large model to obtain the corresponding operator fusion strategy includes: taking the model data, weight data, and hardware features as input, calling a supervised learning model to make operator fusion decisions to obtain the corresponding operator fusion strategy. The supervised learning model is a machine learning model trained based on historical model data, weight data, hardware features, and operator fusion strategies. It minimizes a preset optimization objective by analyzing hardware features and the constraints of the current large model's model data and weight data, and outputs an operator fusion strategy containing at least one fusion operator definition and corresponding execution parameters. The preset optimization objective includes at least one dimension among execution time, resource consumption, hardware adaptability, reusability, or dynamic scene adaptability.

[0101] Based on the aforementioned scheme, in some implementations of this application, the model data includes operator features and dependency graphs; the step of making operator fusion decisions based on the acquired hardware features, as well as the model data and weight data of the local large model, to obtain the corresponding operator fusion strategy includes: using operator features, dependency graphs, and weight data as inputs, performing feature extraction using a graph neural network model to obtain the embedding vector of each operator and the dependency matrix representing the topological structure between operators; inputting the embedding vectors, dependency matrices, and hardware features into a gradient boosting decision tree model, which processes the data according to preset multi-dimensional selection criteria to obtain the corresponding operator fusion strategy, wherein the multi-dimensional selection criteria include at least one of execution time optimization, resource consumption reduction, and hardware adaptability.

[0102] Based on the aforementioned scheme, in some implementations of this application, the step of adjusting the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain the output lexical vector includes: mapping the input lexical vector to a predefined shape bucket according to the sequence length of the input lexical vector; obtaining the fusion operator configuration instantiated by the operator fusion strategy corresponding to the shape bucket; and grouping and / or splitting the input lexical vector according to the block parameters in the fusion operator configuration to obtain the corresponding output lexical vector.

[0103] Based on the aforementioned scheme, in some implementations of this application, the step of generating corresponding computation pipelines and bound resource groups using the large model after operator fusion according to the operator fusion strategy includes: determining whether there are historical computation pipeline records based on the GPU buffer and the identification information of the large model. If they exist, the computation pipelines and bound resource groups in the historical records are reused; if they do not exist, the computation logic of the large model processed according to the operator fusion strategy is compiled into a WebGPU computation shader, and a new computation pipeline is created and bound resource groups are created based on the computation shader.

[0104] Based on the aforementioned scheme, in some implementations of this application, the step of loading the acquired weight data and output lexical vector into the GPU buffer to perform the inference task includes: loading the weight data and output lexical vector into the GPU buffer, and recording GPU instructions through a command encoder to generate a command buffer; submitting the command buffer to the GPU command queue to schedule the computing pipeline to execute the inference task in parallel.

[0105] Based on the aforementioned scheme, in some implementations of this application, after the step of loading the weight data and output lexical vectors into the GPU buffer, the method further includes: loading the weight data and output lexical vectors into the GPU buffer to obtain a queue of tasks to be inferred, wherein each task to be inferred in the queue is a computational task containing each corresponding output lexical vector and computational rule, and to be executed in parallel by the GPU; performing computational task complexity quantification processing based on the weight data, each output lexical vector, and the hardware features to obtain a performance index; and determining whether the performance index corresponding to each task to be inferred exceeds a preset threshold, and if so, decomposing the task to be inferred into multiple fine-grained computational units to be executed in parallel by GPU thread groups based on a block-based strategy.

[0106] Exemplary electronic devices Please see Figure 4 This application provides an electronic device including at least one processor 201 and at least one memory 202. The processor 201 and memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory 202 stores program instructions executable by the processor 201, which can call and execute the program instructions to implement a WebGPU-based large model inference optimization method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, implementing: In response to a smart interaction request, the WebGPU environment is initialized and the corresponding hardware features are acquired. A local large model is obtained, and operator fusion decisions are made based on the hardware features, model data, and weight data of the acquired local large model to obtain a corresponding operator fusion strategy. User input information is acquired, and the large model is used to perform lexicalization on the user input information to obtain an input lexical vector. The granularity of the input lexical vector is adjusted according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector. A GPU buffer is created, and the corresponding computing pipeline and bound resource group are generated using the large model after operator fusion according to the operator fusion strategy. The weight data and output lexical vector are loaded into the GPU buffer, and the inference result is obtained.

[0107] The memory 202 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0108] The processor 201 can be an integrated circuit chip with signal processing capabilities. The processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0109] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0110] Exemplary computer-readable storage media and computer program products This application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor 201, the computer program implements a WebGPU-based large model inference optimization method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, it implements: In response to a browser request to access the physical GPU adapter, a device is created and hardware features are acquired. These hardware features are obtained after the browser initializes the WebGPU environment. The device provides a local computing environment for large model inference. A local large model is acquired, and operator fusion decisions are made based on the acquired hardware features, model data, and weight data of the local large model to obtain a corresponding operator fusion strategy. User input information is acquired, and the large model is used to perform lexicalization on the user input information to obtain an input lexical vector. The granularity of the input lexical vector is adjusted according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector. The WebGPU API interface is called to create a GPU buffer, and the corresponding computing pipeline is generated and bound to a resource group according to the large model computing logic processed by the operator fusion strategy. The WebGPU queue write interface is called to load the acquired weight data and output lexical vector into the GPU buffer and execute to obtain the inference result.

[0111] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] Furthermore, embodiments of this application can also be computer program products, comprising computer program instructions that, when executed by a processor, implement the steps of a WebGPU-based large model inference optimization method according to various embodiments of this application as described in the "Exemplary Methods" section above. For example, implementing: In response to a browser request to access the physical GPU adapter, a device is created and hardware features are acquired. These hardware features are obtained after the browser initializes the WebGPU environment. The device provides a local computing environment for large model inference. A local large model is acquired, and operator fusion decisions are made based on the acquired hardware features, model data, and weight data of the local large model to obtain a corresponding operator fusion strategy. User input information is acquired, and the large model is used to perform lexicalization on the user input information to obtain an input lexical vector. The granularity of the input lexical vector is adjusted according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector. The WebGPU API interface is called to create a GPU buffer, and the corresponding computing pipeline is generated and bound to a resource group according to the large model computing logic processed by the operator fusion strategy. The WebGPU queue write interface is called to load the acquired weight data and output lexical vector into the GPU buffer and execute to obtain the inference result.

[0113] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0114] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A large model inference optimization method based on WebGPU, characterized in that, Includes the following steps: In response to a browser's request to access the physical GPU adapter, a device is created and its hardware characteristics are obtained. These hardware characteristics are those obtained after the browser initializes the WebGPU environment. The device provides a local computing environment for large model inference. A local large model is acquired, and an operator fusion decision is made based on the acquired hardware features, model data and weight data of the local large model to obtain the corresponding operator fusion strategy. The system acquires user input information and uses the large model to perform lexicalization on the user input information to obtain an input lexical vector. The system then adjusts the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector. The WebGPU API is called to create a GPU buffer, and the corresponding computing pipeline is generated and bound to the resource group according to the computing logic of the large model after the operator fusion strategy. The WebGPU queue write interface is called to load the acquired weight data and output word vectors into the GPU buffer, and the inference results are obtained.

2. The method according to claim 1, characterized in that, In the step of creating a device and obtaining hardware characteristics in response to a browser's request to access the physical GPU adapter, the step of obtaining the hardware characteristics includes: The browser requests the GPU adapter and device to initialize the WebGPU environment; The browser queries the maximum workgroup storage size, maximum workgroup size, and device architecture information from the obtained GPU device object to use as hardware characteristics and transmits them to the large model inference service.

3. The method according to claim 1, characterized in that, The step of making operator fusion decisions based on the acquired hardware features, model data of the local large model, and weight data to obtain the corresponding operator fusion strategy includes: The model data, weight data, and hardware features are used as inputs, and the supervised learning model is invoked to make operator fusion decisions in order to obtain the corresponding operator fusion strategy. The supervised learning model is a machine learning model trained based on historical model data, weight data, hardware features, and operator fusion strategies. It minimizes a preset optimization objective by analyzing hardware features and constraints on the model data and weight data of the current large model, and outputs an operator fusion strategy that includes at least one fusion operator definition and corresponding execution parameters. The preset optimization objective includes at least one dimension among execution time, resource consumption, hardware adaptability, reusability, or dynamic scene adaptability.

4. The method according to claim 1, characterized in that, The model data includes operator features and dependency graphs; the step of making operator fusion decisions based on the acquired hardware features, local large model model data, and weight data to obtain the corresponding operator fusion strategy includes: Using operator features, dependency graphs, and weight data as input, a graph neural network model is used to extract features, resulting in the embedding vector of each operator and the dependency matrix representing the topological structure between operators. The embedding vector, dependency matrix, and hardware features are input into the gradient boosting decision tree model, which processes them according to a preset multi-dimensional selection criterion to obtain the corresponding operator fusion strategy. The multi-dimensional selection criterion includes at least one of execution time optimization, resource consumption reduction, and hardware adaptability.

5. The method according to claim 1, characterized in that, The step of adjusting the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain the output lexical vector includes: Based on the sequence length of the input word vector, it is mapped to a predefined shape bucket; Obtain the fusion operator configuration corresponding to the shape bucketing, which is instantiated by the operator fusion strategy; Based on the block parameters in the fusion operator configuration, the input word vector is grouped and / or split to obtain the corresponding output word vector.

6. The method according to claim 1, characterized in that, The step of generating corresponding computing pipelines and binding resource groups based on the computational logic of the large model processed by the operator fusion strategy includes: Based on the GPU buffer and the identification information of the large model, determine whether there are any historical computation pipeline records; If they exist, reuse the compute pipelines and bound resource groups from the history; If it does not exist, the computational logic of the large model after processing according to the operator fusion strategy will be compiled into a WebGPU computation shader, and a new computation pipeline will be created based on the computation shader and a resource group will be bound.

7. The method according to claim 1, characterized in that, The steps of loading the acquired weight data and output word vectors into the GPU buffer and obtaining the inference result include: The weight data and output word vectors are loaded into the GPU buffer, and GPU instructions are recorded by the command encoder to generate the command buffer; The command buffer is submitted to the GPU command queue to schedule the computing pipeline to execute the inference task in parallel and obtain the corresponding inference result.

8. The method according to claim 7, characterized in that, After loading the weight data and output word vectors into the GPU buffer, the process further includes: The weight data and output word vectors are loaded into the GPU buffer to obtain a queue of tasks to be inferred. Each task in the queue is a computation task that contains each corresponding output word vector and computation rule and is to be executed in parallel by the GPU. Based on the weight data, each output word vector, and the hardware features, the computational task complexity is quantified to obtain performance metrics. Based on the performance metrics corresponding to each task to be inferred, it is determined whether a preset threshold is exceeded. If so, the task to be inferred is decomposed into multiple fine-grained computing units that are executed in parallel by GPU thread groups based on a block-based strategy.

9. A large model inference optimization system based on WebGPU, characterized in that, The system includes a browser unit, a large model inference service unit, and a WebGPU interface unit; The browser unit is configured to: in response to a browser's request to access the physical GPU adapter, create a device and obtain hardware characteristics, wherein the hardware characteristics are obtained after the browser initializes the WebGPU environment; the device provides a local computing environment for large model inference; The large model inference service unit, deployed in the browser unit, is configured to: acquire a local large model, and perform operator fusion decisions based on hardware features obtained from the WebGPU interface unit, as well as the model data and weight data of the acquired local large model, to obtain a corresponding operator fusion strategy; utilize the large model to acquire and perform lexical processing on the acquired user input information to obtain an input lexical vector, and adjust the granularity of the input lexical vector according to the operator fusion strategy and the input shape of the input lexical vector to obtain an output lexical vector; call the WebGPU interface unit to create a GPU buffer, and generate corresponding computing pipelines and bind resource groups according to the computing logic of the large model processed by the operator fusion strategy; The WebGPU interface unit is configured to: provide hardware features to the large model inference service unit; in response to a call from the large model inference service unit, perform operations such as creating a GPU buffer, generating a computing pipeline and binding resource groups; and schedule the computing pipeline to perform parallel inference tasks on the weight data and output lexical vectors loaded into the GPU buffer, and return the obtained inference results to the large model inference service.

10. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the one or more programs are executed by the processor, the method as described in any one of claims 1-8 is implemented.

Images