Model inference method, apparatus, and electronic device

Abstract

The present disclosure provides a model inference method and device and electronic equipment, relates to the technical field of computers, in particular to the technical fields of artificial intelligence, large models, front ends, data processing, memory and video memory resource management, and the like. The specific implementation scheme is as follows: in response to an inference request, model data of an inference model is obtained, and the model data is loaded into a memory buffer; wherein the model data includes model structure metadata and weight data; according to the model structure metadata that has been loaded into the memory buffer, video memory resources required by the inference model are created, and the weight data loaded into the memory buffer is written into the video memory resources; the weight data in the video memory resources is read by a graphics processing unit, and the calculation logic of the inference model is executed to obtain an inference result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, large models, front-end, data processing, memory and video memory resource management, and especially to a model inference method, apparatus and electronic device. Background Technology

[0002] With the continuous iteration and deep evolution of edge AI (Artificial Intelligence) technology, the demand for localized AI model deployment on mobile devices has exploded. Driven by the intelligent upgrade of smart terminal devices, diverse AI functions, including LLM (Large Language Model) natural language processing, deep learning-based image recognition, OCR (Optical Character Recognition), and high-precision portrait segmentation, are expected to achieve real-time operation on the edge. However, the current implementation path of mobile AI inference is still dominated by native acceleration frameworks. While these frameworks are efficient in specific scenarios, they have limitations such as insufficient cross-platform compatibility and high development and integration costs. Against this backdrop, WebView, as a mainstream hybrid development container in the field of mobile application development, is becoming a key link in facilitating the deployment of edge AI applications due to its cross-platform characteristics, standardized interfaces, and enhanced built-in model inference capabilities. Summary of the Invention

[0003] This disclosure provides a model reasoning method, apparatus, and electronic device.

[0004] According to one aspect of this disclosure, a model inference method is provided, applied to a WebView container. The method includes: in response to an inference request, obtaining model data of an inference model and loading the model data into a memory buffer; wherein the model data includes model structure metadata and weight data; based on the model structure metadata loaded into the memory buffer, creating video memory resources required by the inference model, and writing the weight data loaded into the memory buffer into the video memory resources; reading the weight data in the video memory resources through a GPU (Central Processing Unit) and executing the computational logic of the inference model to obtain an inference result.

[0005] According to another aspect of this disclosure, a model inference method is provided, applied to the native code layer. The method includes: obtaining an access request for model data of an inference model sent by a WebView container; parsing the access request to obtain raw location information; reading and obtaining model data of the inference model from the local storage area of ​​the web application based on the raw location information; wherein the web application runs in the WebView container; and providing the model data to the WebView container for inference of the inference request.

[0006] According to another aspect of this disclosure, a model inference apparatus is provided, applied to a WebView container. The apparatus includes: an acquisition module, configured to acquire model data of an inference model in response to an inference request; wherein the model data includes model structure metadata and weight data; a loading module, configured to load the model data into a memory buffer; a processing module, configured to create video memory resources required by the inference model based on the model structure metadata loaded into the memory buffer, and write the weight data loaded into the memory buffer into the video memory resources; and an inference module, configured to read the weight data in the video memory resources through the GPU and execute the calculation logic of the inference model to obtain an inference result.

[0007] According to another aspect of this disclosure, a model inference apparatus is provided, applied at the native code layer. The apparatus includes: an acquisition module for acquiring an access request for model data of an inference model sent by a WebView container; a parsing module for parsing the access request to acquire raw location information; a processing module for reading and acquiring model data of the inference model from the local storage area of ​​a web application based on the raw location information; wherein the web application runs in the WebView container; and a providing module for providing the model data to the WebView container for inference of the inference request.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model reasoning method proposed in one aspect of this disclosure, or the model reasoning method proposed in another aspect of this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to execute the model inference method proposed in one aspect of this disclosure, or the model inference method proposed in another aspect of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model reasoning method proposed in one aspect of this disclosure, or the steps of the model reasoning method proposed in another aspect of this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure; Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure; Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 6 This is a block diagram of an electronic device used to implement the model reasoning method of the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] Currently, the main solutions for running AI models within a WebView container are as follows: Option 1. Rely on a native layer inference framework and call it via JSBridge. In this approach, WebView only acts as the display layer and does not directly participate in the inference calculations of the AI ​​model. The actual inference work is performed by the inference framework in the native layer of the mobile device. The WebView container and the native layer inference framework communicate and interact through the JSBridge communication mechanism. JSBridge is a technology that establishes a communication bridge between the WebView container and the native code, allowing JavaScript code in the WebView container to call functions in the native code, while the native code can also send messages to the WebView container. Through JSBridge, the WebView container can send inference requests to the native layer inference framework, passing the input data to the native side. After the native inference framework completes the inference, it returns the result to the WebView container for display via JSBridge.

[0016] However, this scheme has the following characteristics: 1. The WebView container has absolutely no reasoning ability. The WebView container serves merely as a display layer for showing the inference results; all model loading, computation, and inference processes are completed on the native side. This means that WebView itself does not need to have the ability to handle complex model inference.

[0017] 2. Model deployment and updates depend on application version updates or native download capabilities. Since the model performs inference on the native platform, its deployment and updates need to be linked to app version updates or rely on the native platform's download capabilities to obtain new model files. For example, when a model update is needed, it might require releasing a new app version or downloading a new model file from the server via the native platform's download function to replace the existing model. This approach is relatively inflexible, the update cycle can be long, and it cannot achieve real-time or frequent model updates.

[0018] Option 2. WASM (WebAssembly) approach for CPU inference. WASM is a binary instruction set that can run in web browsers. It allows developers to compile code written in languages ​​such as C, C++, and Rust into WASM format and then execute it efficiently in the browser. When running AI models within a WebView container, tools and libraries such as TensorFlow.js, ONNX Runtime Web (WASM), and ggml wasm can be used. These tools and libraries compile AI models into WASM format, enabling the models to run in CPU inference mode within the WebView container.

[0019] For example, TensorFlow.js is a library for training and deploying machine learning models in JavaScript. It supports converting TensorFlow models to WASM format for running in a browser; ONNX Runtime Web (WASM) is an inference runtime based on the ONNX (Open Neural Network Exchange) format. It can compile ONNX models into WASM and perform inference in a WebView container; ggml wasm is a WASM implementation for specific types of models (such as some lightweight language models) that can perform inference computation in a WebView container.

[0020] The limitations of this approach include: 1. Unable to use GPU (Graphics Processing Unit) Most current WASM implementations primarily focus on CPU inference, failing to fully utilize the GPUs of mobile devices for accelerated computation. GPUs offer significant advantages in handling massively parallel computing tasks, greatly improving inference speed. Due to the inability to use GPUs, WASM solutions are severely limited in inference performance.

[0021] 2. Performance is far lower than native. Compared to using a dedicated inference framework on the native side, the WASM solution results in significantly lower performance when running AI models in WebView. This is because native inference frameworks can better integrate with the device's hardware and operating system, fully utilizing the device's underlying optimizations and hardware acceleration capabilities. In contrast, WASM runs in the browser's sandbox environment, which imposes certain limitations and prevents it from achieving native-level performance.

[0022] 3. Model size is limited: The size of WASM modules is limited by browser and device memory. Larger model files can consume significant memory resources during loading and runtime, leading to performance degradation or even memory shortages. Therefore, to ensure smooth operation in WebView, the model size typically needs to be kept within a certain range.

[0023] 4. Unable to load LLM or complex visual models Large language models and complex vision models typically have a huge number of parameters and complex structures, placing high demands on computing resources and memory. Due to the performance and model size limitations of the WASM approach, it cannot meet the needs of these large models, and therefore cannot load and run LLM or complex vision models.

[0024] In summary, the above technologies have the following main shortcomings: 1. The WebView container cannot access local model files and cannot build a large, updatable, and cacheable model system.

[0025] 2. Within the WebView container, AI inference functionality must rely on the computational interfaces provided by the Native layer. While the WebView container provides a runtime environment and system capability interfaces for the H5 page, the H5 page itself cannot independently load and execute models. This tightly coupled architecture not only increases the complexity of hybrid application development but also significantly limits the ability to seamlessly migrate applications across different platforms.

[0026] 3. Edge AI applications lack a unified model management framework, making it difficult to achieve full lifecycle management such as model version verification, incremental updates, and secure deletion.

[0027] 4. Although the WebGPU interface supports GPU-accelerated inference, it cannot directly load large model files in the GB (gigabyte) range due to browser security policies, and the lack of dynamic scheduling mechanism for GPU memory allocation leads to GPU memory overflow or low utilization during large model inference.

[0028] 5. There is permission isolation between the WebView container and the local file system. The browser environment cannot directly operate the sandbox directory, which requires a complex bridging mechanism to transfer model data across layers, seriously affecting the running efficiency of edge AI applications.

[0029] To address at least one of the aforementioned problems, this disclosure proposes a model reasoning method, apparatus, and electronic device.

[0030] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure. It should be noted that the model reasoning method of the present disclosure can be applied to WebView containers.

[0031] like Figure 1 As shown, the model inference method may include the following steps: Step 101: In response to the inference request, obtain the model data of the inference model and load the model data into the memory buffer; wherein, the model data includes model structure metadata and weight data.

[0032] The WebView container can be a browser engine component embedded in a native mobile application or a desktop application.

[0033] Optionally, the inference request may carry information related to the inference model, such as the model identifier, fragment identifier, model version number, etc.

[0034] The inference model can be an AI model that has been trained to perform a specific task. It should be noted that the inference model can be a lightweight model, or a large or even a very large model, etc., and this disclosure does not impose any restrictions on it.

[0035] The model data of the inference model may include model structure metadata and weight data. The model structure metadata can be used to indicate the network structure of the model, such as the type and number of computation layers, the input and output dimensions of the computation layers, the activation function, the size of the convolution kernel, the dependencies between computation layers, etc. This disclosure does not impose any limitations on this. The weight data may include, for example, the value of the convolution kernel, the weight matrix of the fully connected layer, etc.

[0036] The memory buffer can be a pre-allocated storage space in memory used to temporarily store model data.

[0037] In one example, when a user initiates an inference request in the interactive interface of a web application, the WebView container running the web application can capture the inference request and, in response to the inference request, retrieve the model data of the inference model from local storage or the network and load the model data into a memory buffer; the web application can be a web application running in the WebView container, a web application with the ability to call model inference.

[0038] Step 102: Based on the model structure metadata already loaded into the memory buffer, create the video memory resources required for the inference model, and write the weight data loaded into the memory buffer into the video memory resources.

[0039] As an example, after the model data is loaded into the memory buffer, the model structure metadata in the memory buffer can be parsed to determine the memory resource requirements of the inference model. The memory resource requirements may include, for example, the size of the memory space, the type of memory partition, data access permissions, etc. Then, based on the memory resource requirements, the memory resources required for the inference model to execute the inference logic can be created. Finally, the weight data loaded into the memory buffer can be written into the memory resources.

[0040] Step 103: Read the weight data from the GPU memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0041] As an example, the GPU can read weight data from video memory resources and perform various operations sequentially according to the computational logic of the inference model based on the weight data. During the operation, the GPU performs layer-by-layer computation processing on the input data of the inference request that has been synchronously written to the video memory resources and the weight data. The computation results of each computation layer can be temporarily stored in the intermediate buffer of the video memory resources, and finally the inference result is obtained through the output layer operation.

[0042] The model inference method of this disclosure, in response to an inference request, obtains model data of the inference model and loads the model data into a memory buffer; wherein, the model data includes model structure metadata and weight data; based on the model structure metadata loaded into the memory buffer, creates the video memory resources required for the inference model, and writes the weight data loaded into the memory buffer into the video memory resources; the GPU reads the weight data in the video memory resources and executes the calculation logic of the inference model to obtain the inference result. Therefore, the entire model inference process can be completed within the WebView container without relying on the Native layer (referred to as the native code layer in this disclosure). This avoids the communication overhead, compatibility adaptation costs, and data transmission losses caused by cross-Web layer and Native layer calls, and simplifies the implementation path of calling the inference model in the WebView container. At the same time, the model data is loaded in layers through memory buffer transfer, which effectively avoids the compatibility issues of direct data interaction in the WebView container sandbox environment and ensures operational stability. Furthermore, the GPU memory resources are created on demand based on the model structure metadata, and the GPU directly reads the memory weight data to perform calculations, which fully releases the GPU computing power and improves the inference response efficiency. Ultimately, this allows local model inference to be completed efficiently and stably in the pure Web layer, reducing development and deployment costs and meeting the lightweight and highly compatible operation requirements of the WebView container.

[0043] like Figure 2 As shown, this disclosure also proposes a model reasoning method. Figure 2 The schematic diagram is based on the second embodiment of this disclosure. The model reasoning method may include the following steps: Step 201: In response to the inference request, obtain the model data of the inference model; wherein, the model data includes model structure metadata and weight data.

[0044] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0045] In this embodiment of the disclosure, the weight data can be divided into multiple model slices.

[0046] Optionally, in some embodiments, a model download request is sent to the server to obtain model data of the inference model from the server; a data storage request is sent to the native code layer to write the model data in the data storage request to the local storage area of ​​the web application by the native code layer, and to generate an index file corresponding to the inference model.

[0047] The download request may include, but is not limited to, the following: model identifier, model version number, and fragment download identifier. The model identifier can be used to uniquely identify the model, and the fragment download identifier can be used to indicate the fragment data or model fragment to be downloaded from the inference model.

[0048] The web application runs within a WebView container. It's important to note that the WebView container is an independent runtime component or sandbox environment created and managed by the native code layer to host the web application.

[0049] Data storage requests may include, but are not limited to, the following: model data of the inference model, model metadata, the local storage path of the web application, storage permission identifiers, etc. Model metadata may include, for example, model identifiers, shard identifiers, version numbers, etc.

[0050] In one example, the WebView container can send a model download request to the server deploying the inference model. This request can carry authentication information, the model version number, and a chunk download identifier. The server can then parse the request to obtain the authentication information, model version number, and chunk download identifier. After successful server verification, the server can return the inference model matching the model version number and the model data corresponding to the chunk download identifier to the WebView container. Next, the WebView container sends a data storage request to the native code layer via the JSBridge communication mechanism. The native code layer can call system-level file operation interfaces to bypass the WebView container's storage limitations and write the model data to the web application's local storage area. Finally, after storing the model data, the native code layer extracts key information such as storage path, chunk identifier, data size, weight offset, and version number, generates an index file corresponding to the inference model according to a preset format, and can store it in the same directory as the model data. It should be noted that the index file can be used to quickly locate the target chunk data during subsequent model loading without traversing the entire storage file.

[0051] Therefore, by initiating a model download request to the server to obtain inference model data and then writing the model data to the local storage area of ​​the web application using native code, the limitations of browser-side storage capabilities can be overcome. This enables localized persistent storage of inference model data, avoiding the need to re-download model data from the server every time the inference service is used, reducing network transmission overhead and model loading time, and improving the response speed of the inference service. At the same time, the data writing method using native code has higher storage efficiency and stability, ensuring the integrity and accuracy of model data during local storage. The generated inference model index file can help quickly locate key data such as model structure metadata and weight shards in local storage, shortening the data retrieval time during model loading. This allows the web application to directly complete inference calculations based on locally stored model data, providing stable inference services even in weak network or offline environments, thus improving the availability and user experience of the web-based inference service.

[0052] Alternatively, in some embodiments, an access request is sent to the native code layer to retrieve model data from the web application's local storage area via the native code layer.

[0053] The access request may include, but is not limited to, the model identifier of the inference model, the fragment download identifier, the model version number, etc.

[0054] For example, the WebView container can send access requests for model data to the native code layer through the JSBridge communication mechanism. After receiving the access request, the native code layer can parse the access request to obtain the model identifier, the fragment download identifier, the model version number, etc. Then, the native code layer can call the system-level file reading interface to access the local storage area of ​​the web application, locate the corresponding model data according to the model identifier, the model version number and the fragment download identifier, and return it to the WebView container through the callback mechanism.

[0055] Therefore, by sending access requests to the native code layer, which then handles the operation of retrieving model data from the web application's local storage area, the limitations of the browser's access permissions and capabilities to the local file system can be overcome, ensuring stable reading of model data. Compared to web front-end code, the native code layer has higher reading efficiency, which can effectively shorten the reading time of large-volume model data and improve the overall speed of model loading. At the same time, encapsulating the model data reading logic in the native code layer can avoid the security risks caused by the web front-end directly manipulating local storage, such as data leakage and unauthorized tampering, thus maintaining the security and integrity of model data.

[0056] It is understandable that there may be situations where the native code layer fails to read model data. Optionally, in some embodiments, the WebView container can receive exception messages returned by the native code layer and perform exception handling based on the reason for the exception message.

[0057] The error message may include the error code, the reason for the error, etc. The error code can be used to indicate the type of error.

[0058] For example, when the native code layer fails to read model data, it can return an error code and the reason for the exception to the WebView container. Then, after receiving the error code and the reason for the exception, the WebView container can trigger the re-downloading of model data or the exception handling.

[0059] Optionally, in some embodiments, when the weight data is divided into multiple model fragments, in response to the interruption of downloading multiple model fragments, a model download request is resent to the server to obtain the model fragments that have not been downloaded.

[0060] It is understandable that during the process of retrieving model data from the server, there may be situations where model data download is interrupted due to network anomalies, timeouts, server response failures, etc. Therefore, in this embodiment of the disclosure, in the event of an interruption in model data download, the WebView container can resend the model download request to the server to retrieve the incompletely downloaded model data. When the weight data is divided into multiple model fragments, if the download of multiple model fragments is interrupted, the WebView container can resend the model download request to the server to retrieve the incompletely downloaded model fragments.

[0061] The model download request can carry the fragment identifier, model identifier, model version number, etc. of the model fragments that have not been downloaded.

[0062] Specifically, when the download of multiple model segments is interrupted, the WebView container can resend the model download request to the server. After receiving the model download request, the server parses the model download request to obtain the segment identifier, model identifier, model version number, etc. of the model segments that have not been downloaded. Based on this information, the server can locate the data of the model segments that have not been downloaded and return this information to the WebView container.

[0063] Therefore, by re-requesting only the incomplete model fragments instead of the entire inference model data when the download of weighted data fragments is interrupted, the amount of data downloaded repeatedly can be minimized, reducing network bandwidth consumption and overall model download time. This is especially effective in scenarios with a large number of model fragments and large individual fragment sizes, avoiding the waste of resources caused by a single interruption leading to a full re-download. At the same time, this fragment download mechanism with breakpoint resumption can adapt to unstable network environments, improve the fault tolerance and success rate of model data download, and prevent the invalidation of previous download results due to occasional network interruptions, further improving the efficiency and reliability of local storage of web-based inference models.

[0064] Step 202: Load the model structure metadata of the inference model into the memory buffer.

[0065] Step 203: Parse the model structure metadata that has been loaded into the memory buffer to determine the dependencies between multiple computational layers of the inference model and the model slice to which the weight data required by each computational layer belongs.

[0066] Optionally, in any embodiment of this disclosure, the model structure metadata can also be used to indicate the dependencies, sharding mapping rules, etc., between multiple computational layers of the inference model.

[0067] The sharding mapping rule can be used to indicate the model shards corresponding to any computational layer of the inference model.

[0068] As one possible implementation, the WebView container can parse the model structure metadata loaded into the memory buffer to obtain the dependencies between multiple computation layers of the inference model, as well as the sharding mapping rules; then, based on the sharding mapping rules, it can determine the model shard to which the weight data required by any computation layer belongs from multiple model shards.

[0069] Step 204: Based on the dependencies between multiple computing layers, load the model slice corresponding to the computing layer to be executed into the memory buffer.

[0070] Among them, the computation layer to be executed can be a computation layer ordered according to the dependency relationship between multiple computation layers, and all the preceding dependent computation layers have been executed.

[0071] In one example, the computational layers preceding the current computational layer are progressively executed according to the dependencies between multiple computational layers.

[0072] In this embodiment of the disclosure, the current computational layer to be executed can be determined according to the dependencies between multiple computational layers, and the model slice corresponding to the current computational layer to be executed can be loaded into the memory buffer.

[0073] Step 205: Based on the model structure metadata already loaded into the memory buffer, create the video memory resources required for the inference model, and write the weight data loaded into the memory buffer into the video memory resources.

[0074] It should be noted that the execution process of step 205 can refer to the execution process of any embodiment of this disclosure, and will not be described in detail here.

[0075] As one possible implementation, the model structure metadata loaded into the memory buffer is parsed to obtain the computation layer information of the current computation layer to be executed; based on the computation layer information, the weight data required by the current computation layer to be executed is read from the model fragments loaded into the memory buffer; according to the data size of the weight data required by the current computation layer to be executed, the video memory resources required by the current computation layer to be executed are created, and the weight data required by the current computation layer to be executed is written into the corresponding video memory resources.

[0076] The computation layer information may include, but is not limited to: the starting offset address of the weight data in the corresponding model slice, the data size or total number of bytes of the weight data, the storage format of the weight data, the tensor dimension, and the requirements for the video memory storage layout, etc.

[0077] It is understandable that any model slice may include weights of a single computation layer or weights of multiple consecutive computation layers. To improve the utilization of GPU memory resources, for example, the WebView container can first parse the model structure metadata loaded into the memory buffer to obtain the computation layer information of the current computation layer to be executed, such as the starting offset address of the weight data in the corresponding model slice and the total number of bytes of the weight data. Based on the above computation layer information, the WebView container can read the weight data required for the current computation layer to be executed from the binary data of the model slices loaded in the memory buffer according to the starting offset address and data size, without having to read the redundant data of the entire model slice. Then, according to the data size of the read weight data, the container can create the GPU memory resources required for the current computation layer to be executed in the WebGPU by calling the GPU memory creation interface. Finally, the weight data required for the current computation layer to be executed can be written into the GPU memory resources.

[0078] Therefore, by parsing the model structure metadata in the memory buffer to obtain the computational layer information of the computational layer to be executed, the required weight data can be read in a targeted manner from the loaded model fragments, avoiding blind reading of weight data and memory traversal overhead, and ensuring the accuracy and efficiency of data reading. On this basis, video memory resources are created on demand according to the data size of the weight data and the data is written. This avoids the waste of video memory resources due to the preset fixed size of video memory space, and also avoids data writing failure or frequent video memory expansion operations due to insufficient video memory space, realizing the fine-grained and on-demand allocation of video memory resources. At the same time, the weight data is written from the memory buffer to dedicated video memory resources of matching size, reducing the redundant links in the data transfer between memory and video memory, shortening the data migration time, and enabling the GPU to directly access video memory data adapted to its own computing characteristics. This further improves the execution efficiency of the computational layer logic. Especially in inference scenarios where multiple computational layers are executed alternately, it can effectively reduce the overall utilization rate of video memory resources, improve the reuse rate of hardware resources, and thus maintain the resource utilization efficiency and operational stability of the inference process.

[0079] Step 206: Read the weight data from the GPU memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0080] It should be noted that the execution process of step 206 can refer to the execution process of any embodiment of this disclosure, and will not be repeated here.

[0081] As one possible implementation, a shader pipeline corresponding to the current computation layer to be executed is created based on the video memory resources containing the weight data already written to the current computation layer to be executed; the GPU is called through the shader pipeline to read the weight data and / or the output data of the previous computation layer from the video memory resources, and the computation logic of the current computation layer to be executed is executed based on the read data to obtain the output data of the current computation layer to be executed; the inference result is determined based on the output data of multiple computation layers.

[0082] The shader pipeline, also known as the compute shader pipeline, is a dedicated production line for GPUs to perform computations. It can include shader programs (also known as shader code) corresponding to the execution logic of the corresponding compute layer, etc. This disclosure does not limit it.

[0083] For example, the WebView container can invoke its supported WebGPU to implement the following process: Based on the WebGPU's video memory resources corresponding to the currently executed computation layer with written weight data, a shader pipeline corresponding to the currently executed computation layer is created; then, through the shader pipeline, the GPU is invoked, and the GPU accesses the weight data and / or the output data of the previous computation layer in the WebGPU's video memory resources, and executes the computation logic of the currently executed computation layer or its corresponding shader code based on the read data to obtain the output data of the currently executed computation layer; finally, after multiple computation layers have executed their corresponding computation logic and obtained output data, the inference result can be obtained based on the output data of the last layer among the multiple computation layers. For example, the output data of the last layer can be post-processed, such as probability normalization, to obtain the inference result.

[0084] It should be noted that by creating shader pipelines for specific computation layers based on the GPU memory resources containing the weighted data, the data distribution of the shader pipeline and the corresponding computation logic of the computation layer can be precisely matched. This avoids the resource waste and scheduling delays caused by adapting a general pipeline to different computation layers. At the same time, by using the shader pipeline to call the GPU to read the weighted data and / or the output data of the previous layer from the GPU memory resources, the computational advantages of the GPU can be maximized, reducing the data transfer overhead between the CPU and the GPU and improving the execution efficiency of the computation layer logic. In addition, completing the entire process of computation and outputting inference results by executing layer by layer and connecting data ensures the continuity and accuracy of data processing in each computation layer. This not only conforms to the hardware computing characteristics of the GPU, but also makes the computing power scheduling of the inference process more in line with the computation logic of the model, effectively shortening the overall inference time and improving the throughput and response speed of the inference service.

[0085] To clearly illustrate how the shader pipeline corresponding to the currently executing computation layer is created, as one possible implementation, a target shader program is determined from a set of preset shader programs based on the type of the currently executing computation layer; and the shader pipeline corresponding to the currently executing computation layer is created based on the target shader program and the video memory resources containing the weight data already written to the currently executing computation layer.

[0086] The types may include, but are not limited to: multi-head attention computation layer, matrix multiplication computation layer, layer normalization computation layer, positional encoding computation layer, etc., or may include: fully connected layer, attention layer, normalization layer, convolutional layer, positional encoding layer, etc.

[0087] Optionally, in some embodiments, the shader program may include at least one of the following: a multi-head attention shader program, a matrix multiplication shader program, a layer normalization shader program, a position encoding shader program, and a target shader program; wherein the target shader program is used to drive token generation or caching.

[0088] Among them, the multi-head attention shader program can be used to perform the core matrix operations and attention weight calculations of multi-head attention; the matrix multiplication shader program can be used for the basic multiplication and addition operations of fully connected layers or attention layers; the layer normalization shader program can be used to perform the normalization logic of feature data; the position encoding shader program can be used to add position information to the input token, etc.

[0089] It should be noted that shader programs can be pre-configured and can be configured as needed; this disclosure does not impose any restrictions on this.

[0090] Therefore, by pre-setting the shader program system, full-scenario coverage of the core computational links of inference models such as large language models is achieved, ensuring that different types of computational layers can be matched with customized shader programs, avoiding the efficiency loss when adapting general programs to computational logic. At the same time, by decomposing various computational logics into corresponding shader programs, it is easier to perform independent performance tuning and version iteration for different computational links, reducing the maintenance cost of the overall inference framework, and enabling GPU computing power to be accurately released in different computational links, maximizing the utilization of computing resources during inference, and maintaining the execution efficiency and overall synergy of each core computational layer when inferring complex models.

[0091] As an example, the Webview container can identify the type of the current computation layer to be executed. Then, based on the type of the current computation layer to be executed, it can determine the target shader program that is compatible with the current computation layer from a number of preset shader programs. Then, based on the target shader program, it can call the WebGPU it supports, and according to the WebGPU pipeline construction rules, based on the video memory resources that have been written to the weight data of the corresponding slice, etc., to create a shader pipeline that is compatible with the execution logic of the current computation layer to be executed.

[0092] Therefore, matching the target shader program from the preset shader programs based on the type of the current computation layer ensures a high degree of compatibility between the shader program and the computation characteristics and data processing logic of the computation layer. This avoids computational redundancy or functional deficiencies caused by adapting general shader programs to different types of computation layers. Furthermore, by creating a dedicated shader pipeline using the memory resources with written weight data, the pipeline's functional specificity and execution efficiency are further maintained. This eliminates the need for dynamically compiling shader programs to adapt to different computation layers, shortening pipeline creation time. It also enables the GPU to accurately call the appropriate computation instructions when executing computation layer logic, effectively reducing invalid instruction scheduling and wasted computing power, effectively lowering the probability of errors during pipeline operation, and improving the stability and reliability of model inference.

[0093] Optionally, in some embodiments, when the GPU is called through a target interface, before reading the weight data in the video memory resources through the GPU and executing the calculation logic of the inference model to obtain the inference result, the target interface can be detected to determine whether the target interface is available. If the target interface is determined to be available, the GPU is then called through the target interface to read the weight data in the video memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0094] The target interface can be, for example, WebGPU.

[0095] For example, the WebView container can detect whether the WebGPU interface exists, confirm whether the WebGPU interface has calling permissions, and verify whether the WebView container has granted access permissions to the WebGPU interface (i.e., verify whether the WebGPU interface is supported by the WebView container). After all the above checks are passed, an interface availability status flag can be generated. Then, the WebView container can read the interface availability status flag of the WebGPU interface. When the interface availability status flag indicates that the WebGPU interface is available, the WebView container determines that the WebGPU is available and can call the GPU through the WebGPU to read the weight data in the video memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0096] Therefore, by performing availability checks on the target interface before calling the GPU to execute core computing logic, and only triggering the GPU call process when the interface is confirmed to be available, the problem of GPU call failure or computing interruption caused by interface anomalies can be avoided from the source. This avoids invalid operations in subsequent stages such as weight data reading and computing logic execution, reducing the waste of computing resources and time costs. At the same time, as a pre-verification step before inference execution, interface availability checks can expose potential faults at the interface level in advance, making it easier to quickly locate and handle problems, reducing the probability of sudden interruptions during inference, and improving the robustness and fault tolerance of the inference service. In addition, judging the availability status of the interface can ensure the one-time successful execution of the GPU call process, avoiding the increase in inference latency caused by interface anomaly retries. Especially in high-concurrency inference service scenarios, it can effectively maintain the stability and timeliness of inference response and improve the service quality at the business level.

[0097] In one possible implementation of this disclosure, in response to the completion of the computational logic of the currently executing computational layer, the video memory resources and / or memory buffers corresponding to the currently executing computational layer can be released.

[0098] For example, when the WebView container confirms through the WebGPU interface that the shader pipeline of the currently executing computation layer has been completed and the output data of the layer has been written to the video memory resources, it can determine that the computation logic has been completed. Then, the WebView container can release the video memory resources corresponding to the currently executing computation layer through the WebGPU interface, and can set the memory buffer where the weight data corresponding to the currently executing computation layer is written to be empty, triggering the corresponding garbage collection mechanism to reclaim CPU memory.

[0099] Therefore, after the computational logic of the current computational layer is completed, its corresponding video memory resources and / or memory buffers are released in a timely manner, realizing the dynamic reclamation and cyclic reuse of hardware resources. This avoids the long-term idle video memory and memory space occupied by the completed computation layers, improving the overall utilization rate of memory and video memory resources. Especially when dealing with multi-level, large-parameter inference models, this on-demand release mechanism can effectively control the peak of overall resource usage, allowing limited hardware resources to support the alternating execution of more computational layers, adapting to high-concurrency inference service scenarios. At the same time, timely release of unused resources can also reduce the generation of memory fragmentation, reduce the risk of subsequent memory allocation failures caused by resource fragmentation, further improve the stability and continuity of the inference process, and extend the runtime of uninterrupted inference services.

[0100] Optionally, in some embodiments, the inference results are displayed in the interactive interface of a web application.

[0101] As an example, the inference results processed by the GPU can be converted into a user-understandable format. Then, the WebView container can display the processed inference results in the web application's interactive interface. Optionally, the interface's interactive state can be updated synchronously, such as the loading completion status and the results display area. Therefore, displaying the inference results in the web application's interactive interface allows the inference results to reach end users in an intuitive way, lowering the barrier to entry for inference services and improving the user experience.

[0102] The model inference method of this disclosure loads the model structure metadata of the inference model into a memory buffer; parses the loaded model structure metadata in the memory buffer to determine the dependencies between multiple computational layers of the inference model, and the model fragments to which the weight data required by each computational layer belongs; based on the dependencies between multiple computational layers, loads the model fragments corresponding to the computational layer to be executed into the memory buffer. This enables on-demand fragment loading of the inference model, eliminating the need to load all weight data of the entire inference model into memory at once, reducing memory usage costs during model inference, and adapting to inference deployment environments with low memory resources. Simultaneously, loading corresponding model fragments based on the dependencies of computational layers avoids invalid fragment loading and redundant memory usage, reducing data loading time and hardware resource consumption, effectively improving the overall execution efficiency of model inference. Furthermore, through structured metadata parsing and fragment scheduling, memory management for model inference becomes more targeted and controllable, maintaining the smoothness and stability of the inference process, making it particularly suitable for the efficient deployment and operation of complex inference models with large parameters and multiple levels.

[0103] The above describes the model inference method on the WebView container side. This disclosure also proposes a model inference method on the native code layer side.

[0104] like Figure 3 As shown, this disclosure also proposes a model reasoning method. Figure 3 The schematic diagram is based on the third embodiment of this disclosure. The model reasoning method may include the following steps: Step 301: Obtain the access request for model data for the inference model sent by the WebView container.

[0105] It should be noted that the explanations of the inference model, model data, and access requests in any of the above embodiments of this disclosure also apply to this embodiment, and will not be repeated here.

[0106] In this embodiment of the disclosure, the WebView container can send an access request for model data of the inference model to the native code layer, and then the native code layer can receive the access request.

[0107] Step 302: Parse the access request to obtain the original location information.

[0108] The original location information may include, but is not limited to, the model identifier, fragment identifier, model version number, etc. of the inference model.

[0109] In this embodiment of the disclosure, the native code layer can parse the access request to obtain the original location information.

[0110] Step 303: Based on the original location information, read and obtain the model data of the inference model from the local storage area of ​​the web application.

[0111] Web applications can run within a WebView container.

[0112] For example, the native code layer can verify the legality of the original location information, such as verifying whether the model identifier and segment identifier are within the valid range. After the original location information is verified, the native code layer can access the local storage area of ​​the web application and read the model data of the inference model.

[0113] As one possible implementation, local location information corresponding to the original location information is determined according to a preset path mapping rule; model data is then read and obtained from the local storage area of ​​the web application based on the local location information.

[0114] The path mapping rules can be pre-defined.

[0115] As an example, after obtaining the original location information at the native code layer, the local location information corresponding to the original location information can be determined according to the preset path mapping rules. Then, the native code layer can access the local storage area of ​​the web application based on the local location information to locate and read the corresponding model data.

[0116] For example, after obtaining the original location information at the native code layer, the parameters in the original location information can be replaced according to the preset path mapping rules. Then, the replaced parameters are concatenated with paths to generate standardized local location information. Subsequently, the native code layer can call the system-level file reading interface, using the local location information as the file path, to read and obtain model data from the local storage area of ​​the web application.

[0117] Therefore, by using preset path mapping rules, the original location information passed from the WebView container can be converted into local location information that matches the local storage. Model data can then be read based on this information, decoupling the location logic on the WebView container side from the actual path in local storage. The WebView side does not need to be aware of the physical path structure of local storage, improving code compatibility and portability. Simultaneously, the path mapping rules can standardize and convert the original location information, avoiding model data reading failures due to non-standard path formats or incorrect location information, thus improving data reading accuracy. Furthermore, by shielding the real path in local storage through path mapping, data security risks caused by directly exposing the physical path in the original location information can be prevented, strengthening the storage security of local model data. Moreover, the standardized path mapping logic facilitates unified maintenance and adjustment. When the local storage path changes, only the mapping rules need to be modified for adaptation, without adjusting the business code within the WebView container, reducing maintenance costs.

[0118] As one possible implementation, the process of storing the model data of the inference model in the local storage area may include the following steps: receiving a data storage request sent by the WebView container and parsing the data storage request to obtain the model data, wherein the model data is obtained by the WebView container from the server; writing the model data in the data storage request into the local storage area of ​​the web application and generating an index file corresponding to the inference model.

[0119] Data storage requests may include, but are not limited to, the following: model data of the inference model, model metadata, the local storage path of the web application, storage permission identifiers, etc. Model metadata may include, for example, model identifiers, shard identifiers, version numbers, etc.

[0120] The index file may contain information such as model identifier, version number, shard identifier, storage path, and data size.

[0121] For example, after the WebView container obtains the model data of the inference model from the server, it can send a data storage request to the native code layer through the JSBridge communication mechanism. Correspondingly, the native code layer receives the data storage request sent by the WebView container through the JSBridge communication mechanism, parses the data storage request, obtains the model data of the inference model, and writes the model data into the local storage area of ​​the web application. After the data is written, the native code layer can generate the index file corresponding to the inference model based on the model metadata and the actual storage path.

[0122] Therefore, by having the native code layer handle the model data storage requests from the WebView container, parse them, write the model data to the local storage area of ​​the web application, and generate an index file, the high-performance data reading capabilities of the native layer can be leveraged to efficiently complete the local writing of large-volume model data. Compared to the WebView container directly manipulating local storage, this improves data writing efficiency and stability, avoiding storage failures caused by browser sandbox limitations. Simultaneously, having the native layer uniformly responsible for model data storage allows for integrity verification during the data writing process, ensuring the accuracy of model data in local storage and preventing model data corruption due to write interruptions or data packet loss. The generated inference model index file clearly records key information about the model data, providing structured guidance for the WebView container to quickly retrieve and read model data, shortening retrieval time during model loading, and further improving the overall response efficiency of inference services in WebView scenarios. Furthermore, it separates the responsibilities of model data acquisition and storage, reducing the performance footprint of the WebView container and facilitating independent optimization and maintenance of storage logic, thus exhibiting scalability.

[0123] Step 304: Provide the model data to the WebView container for inference of the inference request.

[0124] In this embodiment of the disclosure, the native code layer can return the read model data to the WebView container according to the callback mechanism.

[0125] It is understandable that read failures may occur. Optionally, in some embodiments, when the native code layer fails to read, the native code layer can generate an exception message based on the exception reason and error type, and send the exception message to the WebView container.

[0126] The error message may include the error code, the reason for the error, etc. The error code can be used to indicate the type of error.

[0127] For example, after the WebView container receives an exception message, it can render the exception message to the interactive interface of the web application running in the WebView container to display the exception message to the relevant user, so that the user can understand the specific reason for the reading failure in a timely manner and perform the corresponding exception handling.

[0128] The model inference method of this disclosure includes: obtaining an access request for model data of the inference model sent by the WebView container; parsing the access request to obtain the original location information; reading and obtaining the model data of the inference model from the local storage area of ​​the web application based on the original location information; wherein the web application runs in the WebView container; and providing the model data to the WebView container for inference of the inference request. Therefore, by obtaining the model data access request sent by the WebView container, parsing the original location information, reading the model data from the web application's local storage, and feeding it back to the WebView container, a model data access link between the WebView container and local storage can be established. This avoids access failures or data reading anomalies caused by the container sandbox mechanism and permission restrictions when the WebView container directly accesses local storage, ensuring the stability of model data reading. At the same time, it can achieve targeted reading of model data based on the parsed original location information, avoiding a full traversal of local storage, shortening data retrieval and reading time, and improving the startup speed of the inference service within the WebView container. In addition, this method allows the WebView container to stably obtain model data from local storage and complete inference calculations without relying on network transmission. This reduces network dependence, makes full use of local hardware resources, expands the application boundaries of inference services in WebView scenarios, and improves the user experience for end users.

[0129] To clearly illustrate the model inference method disclosed herein, a detailed explanation is provided below with examples.

[0130] As an example, model inference methods can have the following mechanisms: 1. Model download and local persistence mechanism 1.1 During the operation of the web application, the WebView container calls a remote server to obtain model fragments of the inference model.

[0131] For example, when the web application is running, the WebView container can proactively detect model update notifications pushed by the remote server or proactively query to obtain new version information of the inference model. Subsequently, it can call the remote server to download the updated model fragments.

[0132] 1.2 The WebView container interacts with the Native layer (referred to as the native code layer in this disclosure) through the JSBridge communication mechanism, and writes the obtained model fragment data into the local storage area corresponding to the real path of the private directory of the web application through the Native layer.

[0133] 1.3 After the model shards are written, the Native layer generates the index file of the inference model.

[0134] The index file includes the version number of the inference model, hash value, number of shards, etc. The hash value is used to verify data integrity, ensuring the accuracy of the downloaded shard data.

[0135] It should be noted that this implementation method can bypass the limitation that WebView cannot directly write files, enabling the storage of model files in the web environment; it can support the function of resuming interrupted downloads of large models, so that when the download is interrupted, it can continue downloading from the point of interruption, improving download efficiency; at the same time, it supports hot updates of models, which can quickly update models without relying on application releases, providing users with more timely and high-quality services.

[0136] 2. Virtual path mapping mechanism for local model files 2.1 The Native layer can pre-create the access mapping mechanism of the local inference model (referred to as path mapping rules in this disclosure) to define the conversion rules between private directory paths and secure URLs (Uniform Resource Locator). 2.2 The Native layer can map the real path of the private directory of the web application where the model file of the inference model is located to a URL that can be recognized by the WebView container and conforms to security specifications, according to the conversion rules.

[0137] Optionally, in some embodiments, when the browser kernel built into the WebView container initiates an access request to the secure URL, the Native layer can intercept the request and perform a validity check, such as checking whether the request source is the current application and whether the access permission is compliant. If the check passes, the URL is routed to the corresponding private directory real path; if the check fails, access is denied.

[0138] Alternatively, in some embodiments, the conversion rules may use local file access protocols (i.e., file: / / ), content access protocols (i.e., content: / / ), custom URL protocols, or construct virtual file directories through the Native layer, etc.

[0139] It should be noted that, through the above mapping mechanism, this system can have the following advantages: 1) It breaks through the limitation that the WebView container can only access limited resources within the sandbox, and gives the WebView container the ability to access local model files (referred to as model data in this disclosure) in the application's private directory, without having to copy the model files to the WebView sandbox, thus reducing data redundancy; 2) The model file is always stored in the application's private directory, which other applications do not have access to. All URL access requests must be validated by the native layer. This satisfies the flexibility of web applications to read model data, while eliminating the risk of model data leakage from the bottom layer. 3) By routing paths directly through the Native layer, frequent JSBridge communication between the WebView container and the native layer is avoided, thus improving the efficiency of accessing model files.

[0140] 3. WebGPU Weighted Streaming Loading and Memory Management Mechanism 3.1. The WebView container can read the model file piece by piece as an ArrayBuffer (referred to as the memory buffer in this disclosure). 3.2 The WebView container parses the model structure of the inference model, determines the model fragment to which the weight data required by the current computation layer belongs, extracts the corresponding data from the ArrayBuffer, creates a Buffer through the WebGPU interface, and uploads the extracted data to the video memory resource. 3.3 Based on the aforementioned video memory resources, the WebView container constructs the shader pipeline corresponding to the current computation layer to be executed through the WebGPU interface, and calls the GPU through the shader pipeline to read the weight data and / or the output data of the previous computation layer in the video memory resources. Based on the read data, it executes the computation logic of the current computation layer to be executed and obtains the output data of the current computation layer to be executed. Based on the output data of multiple computation layers, it determines the inference result.

[0141] It should be noted that, in response to the completion of the computation logic of the currently executing computation layer, the corresponding video memory resources and / or ArrayBuffer of the currently executing computation layer are released.

[0142] It should also be noted that, through the above mechanism, this system can have the following advantages: 1) Enable loading and inference of large models ranging from hundreds of MB to GB in the WebGPU environment; 2) The on-demand loading and automatic release mechanism can effectively reduce video memory usage and prevent the device from lag or crashing due to resource exhaustion; 3) Piece loading instead of full loading can reduce model startup time; 4) Solved the technical problem that WebGPU is limited by memory or video memory and cannot load extremely large weighted files at once.

[0143] Optionally, the shader program corresponding to the shader pipeline may include at least one of the following: multi-head attention shader program, matrix multiplication shader program, layer normalization shader program, position encoding shader program, and target shader program; wherein, the target shader program is used to drive token generation or caching.

[0144] Therefore, it can provide GPU inference performance similar to that of the native layer within the WebView container, and can also be compatible with the operation of multiple types of models such as large language models, visual models, and OCR sub-models.

[0145] 4. Inference scheduling and result feedback mechanism 4.1 Before calling the GPU via WebGPU for model inference, the WebView container can detect the availability of WebGPU in the current environment; 4.2 If WebGPU is detected to be available, the model inference is performed by calling the GPU through WebGPU, and the GPU output results are sent back to the business layer; 4.3 If WebGPU is detected to be unavailable, it will automatically downgrade to WASM (WebAssembly) or the Native layer inference path to maintain uninterrupted inference functionality.

[0146] Therefore, an adaptive cross-platform inference strategy was designed, prioritizing WebGPU and using WASM or Native as a fallback, which can adapt to the computing power of different devices without manual intervention; for devices with weaker capabilities or no WebGPU, the inference function can be maintained to run normally through a degradation mechanism, with full device compatibility.

[0147] Optionally, in some embodiments, the WebGPU inference path can be omitted during model inference. Instead, WASM (WebAssembly) combined with SIMD (Single Instruction Multiple Data) technology can be used to implement model inference. Specifically, the WebView container can compile the inference operators of the model into a WASM module that supports SIMD. Then, the WebView container can load the compiled WASM module, initialize the runtime environment, write the weight data of the inference model into the linear memory accessible by WASM, and call the operator functions in the WASM module to complete the calculation logic of the inference model and obtain the inference result.

[0148] Inference operators can include, for example, multi-head attention operators, matrix multiplication (MatMul) operators, and layer normalization operators.

[0149] Linear memory can be a contiguous byte buffer shared by the WebView container and WASM.

[0150] Optionally, in some embodiments, when the inference model is a small to medium-sized model, the model can be downloaded as a whole.

[0151] Optionally, in the local model inference scenario based on the WebView container, the original WebGPU inference implementation can be replaced by using the ONNX Runtime WebGPU engine or the WebLLM engine to perform inference work.

[0152] Optionally, the computational operations during the WebGPU inference process can be migrated from the UI thread to the WebWorker background thread, whereby the WebWorker thread schedules the GPU to complete the model inference computation process.

[0153] In summary, the model inference method disclosed herein has at least the following advantages: 1. Adopting a full-endpoint AI deployment model eliminates reliance on the server. This deployment method, on the one hand, eliminates the need to build and maintain a complex server architecture, effectively reducing deployment costs; on the other hand, the model can run continuously on the user's local device, avoiding the high costs of purchasing, deploying, and using GPU servers over a long period, thus saving resource investment.

[0154] 2. Successfully overcame technical limitations, natively supporting local inference of large language models within the WebView environment. This means that users can directly perform large language model inference operations in common web applications without the need for additional plugins or tools, expanding the functional boundaries of WebView and bringing users a more convenient interactive experience.

[0155] 3. It possesses powerful model hot update capabilities, eliminating the need for application re-releases. Previously, large model iterations were limited by application release cycles, resulting in cumbersome and time-consuming update processes. This technology allows for rapid model iteration; once a new version is available, it can be promptly pushed to users, enabling quick deployment and ensuring users have access to the latest model features immediately.

[0156] 4. In terms of user experience, since the inference process is performed locally, network transmission is reduced, resulting in a significant improvement in response speed. It also supports offline use, unaffected by network conditions. Furthermore, leveraging WebGPU technology, it fully utilizes the hardware performance of local devices, achieving near-native edge-side inference performance and providing users with a smooth and efficient experience.

[0157] To implement the above embodiments, this disclosure also provides a model inference apparatus. For example... Figure 4 As shown, Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. The model inference device 400 may include: an acquisition module 401, a loading module 402, a processing module 403, and an inference module 404.

[0158] The acquisition module 401 is used to acquire model data of the inference model in response to the inference request; wherein the model data includes model structure metadata and weight data.

[0159] Loading module 402 is used to load model data into the memory buffer.

[0160] The processing module 403 is used to create the video memory resources required by the inference model based on the model structure metadata loaded into the memory buffer, and write the weight data loaded into the memory buffer into the video memory resources.

[0161] The inference module 404 is used to read weight data from the GPU memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0162] In one possible implementation of this disclosure, the weight data is divided into multiple model fragments; the loading module 402 is used to: load the model structure metadata of the inference model into a memory buffer; parse the model structure metadata loaded into the memory buffer to determine the dependencies between multiple computation layers of the inference model, and the model fragment to which the weight data required by each computation layer belongs; and load the model fragment corresponding to the current computation layer to be executed into the memory buffer based on the dependencies between multiple computation layers.

[0163] In one possible implementation of this disclosure, the processing module 403 is configured to: parse the model structure metadata loaded into the memory buffer to obtain the computation layer information of the current computation layer to be executed; based on the computation layer information, read the weight data required by the current computation layer to be executed from the model fragments loaded into the memory buffer; create the video memory resources required by the current computation layer to be executed according to the data size of the weight data required by the current computation layer to be executed, and write the weight data required by the current computation layer to the corresponding video memory resources.

[0164] In one possible implementation of this disclosure, the inference module 404 is configured to: create a shader pipeline corresponding to the current computational layer to be executed based on the video memory resources containing weight data already written to them; call the GPU through the shader pipeline to read the weight data and / or the output data of the previous computational layer from the video memory resources, and execute the computational logic of the current computational layer to be executed based on the read data to obtain the output data of the current computational layer to be executed; and determine the inference result based on the output data of multiple computational layers.

[0165] In one possible implementation of this disclosure, the inference module 404 is configured to: determine a target shader program from a plurality of preset shader programs according to the type of the current computation layer to be executed; and create a shader pipeline corresponding to the current computation layer to be executed based on the target shader program and the video memory resources with written weight data corresponding to the current computation layer to be executed.

[0166] In one possible implementation of this disclosure, the shader program includes at least one of the following: a multi-head attention shader program, a matrix multiplication shader program, a layer normalization shader program, a position encoding shader program, and a target shader program; wherein the target shader program is used to drive token generation or caching.

[0167] In one possible implementation of this disclosure, the model inference device 400 may further include: The release module is used to release the video memory resources and / or memory buffers corresponding to the currently executing computational layer in response to the completion of the computational logic of the currently executing computational layer.

[0168] In one possible implementation of this disclosure, the model inference device 400 may further include: The first sending module is used to send a model download request to the server in order to obtain the model data of the inference model from the server.

[0169] The second sending module is used to send data storage requests to the native code layer so that the model data in the data storage request is written from the native code layer to the local storage area of ​​the web application, and an index file corresponding to the inference model is generated.

[0170] In one possible implementation of this disclosure, the acquisition module 401 is configured to: send an access request to the native code layer to obtain model data from the local storage area of ​​the web application through the native code layer; the web application runs in a WebView container.

[0171] In one possible implementation of this disclosure, the weight data is divided into multiple model fragments, and the first sending module is further configured to: in response to the interruption of downloading multiple model fragments, resend a model download request to the server to obtain the model fragments that have not been downloaded.

[0172] In one possible implementation of this disclosure, the GPU is invoked through a target interface. The inference module 404 is used to: detect the target interface to determine whether the target interface is available; if the target interface is determined to be available, invoke the GPU through the target interface to read the weight data in the video memory resources and execute the calculation logic of the inference model to obtain the inference result.

[0173] In one possible implementation of this disclosure, the model inference device 400 may further include: The presentation module is used to display the reasoning results in the interactive interface of the web application; the web application runs in a WebView container.

[0174] It should be noted that the model inference apparatus provided in this embodiment is capable of achieving the above-mentioned... Figures 1 to 2 All the method steps implemented in the model reasoning method embodiment are the same as those in the method embodiment, and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail here.

[0175] The model inference apparatus of this disclosure, in response to an inference request, acquires model data of an inference model and loads the model data into a memory buffer; wherein, the model data includes model structure metadata and weight data; based on the model structure metadata loaded into the memory buffer, it creates the video memory resources required for the inference model and writes the weight data loaded into the memory buffer into the video memory resources; it reads the weight data in the video memory resources through the GPU and executes the calculation logic of the inference model to obtain the inference result. Therefore, the entire model inference process can be completed within the WebView container without relying on the Native layer (referred to as the native code layer in this disclosure). This avoids the communication overhead, compatibility adaptation costs, and data transmission losses caused by cross-Web layer and Native layer calls, and simplifies the implementation path of calling the inference model in the WebView container. At the same time, the model data is loaded in layers through memory buffer transfer, which effectively avoids the compatibility issues of direct data interaction in the WebView container sandbox environment and ensures operational stability. Furthermore, the GPU memory resources are created on demand based on the model structure metadata, and the GPU directly reads the memory weight data to perform calculations, which fully releases the GPU computing power and improves the inference response efficiency. Ultimately, this allows local model inference to be completed efficiently and stably in the pure Web layer, reducing development and deployment costs and meeting the lightweight and highly compatible operation requirements of the WebView container.

[0176] To implement the above embodiments, this disclosure also provides a model inference apparatus. For example... Figure 6 As shown, Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure. The model inference device 500 may include: an acquisition module 501, a parsing module 502, a processing module 503, and a providing module 504.

[0177] The acquisition module 501 is used to acquire the access request for model data for the inference model sent by the WebView container.

[0178] The parsing module 502 is used to parse the access request to obtain the original location information.

[0179] The processing module 503 is used to read and obtain model data of the inference model from the local storage area of ​​the web application based on the original location information; wherein the web application runs in the WebView container.

[0180] Module 504 is provided to provide model data to the WebView container for inference of inference requests.

[0181] In one possible implementation of this disclosure, the processing module 503 is configured to: determine local location information corresponding to the original location information according to a preset path mapping rule; and read and obtain model data from the local storage area of ​​the web application according to the local location information.

[0182] In one possible implementation of this disclosure, the model inference device 500 may further include: The receiving module is used to receive data storage requests sent by the WebView container.

[0183] The parsing module is used to parse data storage requests to obtain model data, which is obtained by the WebView container from the server.

[0184] The write module is used to write model data from data storage requests to the local storage area of ​​the web application.

[0185] The generation module is used to generate the index file corresponding to the inference model.

[0186] It should be noted that the model inference apparatus provided in this embodiment is capable of achieving the above-mentioned... Figure 3 All the method steps implemented in the model reasoning method embodiment are the same as those in the method embodiment, and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail here.

[0187] The model inference apparatus of this disclosure acquires an access request for model data of an inference model sent by a WebView container; parses the access request to obtain original location information; and reads and acquires the model data of the inference model from the local storage area of ​​the web application based on the original location information; wherein the web application runs in a WebView container; and provides the model data to the WebView container for inference of the inference request. Therefore, by obtaining the model data access request sent by the WebView container, parsing the original location information, reading the model data from the web application's local storage, and feeding it back to the WebView container, a model data access link between the WebView container and local storage can be established. This avoids access failures or data reading anomalies caused by the container sandbox mechanism and permission restrictions when the WebView container directly accesses local storage, ensuring the stability of model data reading. At the same time, it can achieve targeted reading of model data based on the parsed original location information, avoiding a full traversal of local storage, shortening data retrieval and reading time, and improving the startup speed of the inference service within the WebView container. In addition, this method allows the WebView container to stably obtain model data from local storage and complete inference calculations without relying on network transmission. This reduces network dependence, makes full use of local hardware resources, expands the application boundaries of inference services in WebView scenarios, and improves the user experience for end users.

[0188] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, are all carried out with the user's consent and comply with relevant laws and regulations, and do not violate public order and good morals.

[0189] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0190] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0191] like Figure 6As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0192] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0193] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as model inference methods. For example, in some embodiments, the model inference method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the model inference method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform model inference methods by any other suitable means (e.g., by means of firmware).

[0194] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0195] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0196] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0197] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0198] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0199] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0200] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model reasoning method applied to a WebView container, the method comprising: In response to an inference request, the model data of the inference model is obtained and loaded into a memory buffer; wherein, the model data includes model structure metadata and weight data; Based on the model structure metadata already loaded into the memory buffer, create the video memory resources required for the inference model, and write the weight data loaded into the memory buffer into the video memory resources; The weight data in the video memory resources is read by the graphics processor, and the calculation logic of the inference model is executed to obtain the inference result.

2. The method according to claim 1, wherein, The weight data is divided into multiple model slices; Loading the model data into the memory buffer includes: Load the model structure metadata of the inference model into the memory buffer; The model structure metadata loaded into the memory buffer is parsed to determine the dependencies between multiple computational layers of the inference model, and the model slice to which the weight data required by each computational layer belongs; Based on the dependencies between the multiple computing layers, the model slice corresponding to the computing layer to be executed is loaded into the memory buffer.

3. The method according to claim 2, wherein, The step of creating the GPU memory resources required for the inference model based on the model structure metadata already loaded into the memory buffer, and writing the weight data loaded into the memory buffer into the GPU memory resources, includes: The model structure metadata loaded into the memory buffer is parsed to obtain the computation layer information of the current computation layer to be executed. Based on the computation layer information, the weight data required for the current computation layer to be executed is read from the model slices that have been loaded into the memory buffer; Based on the data size of the weight data required by the current computation layer to be executed, create the video memory resources required by the current computation layer to be executed, and write the weight data required by the current computation layer to be executed into the corresponding video memory resources.

4. The method according to claim 3, wherein, The step of reading weight data from the graphics memory resources through the graphics processor and executing the calculation logic of the inference model to obtain the inference result includes: Based on the video memory resources containing the weight data already written to the current computation layer to be executed, create the shader pipeline corresponding to the current computation layer to be executed. The graphics processor is invoked through the shader pipeline to read the weight data and / or the output data of the previous computing layer in the video memory resources, and the computing logic of the current computing layer to be executed is executed based on the read data to obtain the output data of the current computing layer to be executed. The inference result is determined based on the output data of the multiple computing layers.

5. The method according to claim 4, wherein, The step of creating the shader pipeline corresponding to the currently executing computation layer based on the video memory resources containing the weighted data already written to it includes: Based on the type of the current computation layer to be executed, a target shader program is determined from a plurality of preset shader programs; Based on the target shader program and the video memory resources containing the weight data already written to the current computation layer to be executed, create the shader pipeline corresponding to the current computation layer to be executed.

6. The method according to claim 5, wherein, The shader program includes at least one of the following: The program includes a multi-head attention shader program, a matrix multiplication shader program, a layer normalization shader program, a position encoding shader program, and a target shader program; wherein the target shader program is used to drive token generation or caching.

7. The method according to claim 4, wherein, The method further includes: In response to the completion of the computation logic of the currently executing computation layer, the video memory resources and / or the memory buffer corresponding to the currently executing computation layer are released.

8. The method according to claim 1, wherein, The method further includes: Send a model download request to the server to obtain the model data of the inference model from the server; A data storage request is sent to the native code layer to write the model data in the data storage request to the local storage area of ​​the web application and generate an index file corresponding to the inference model; the web application runs on the WebView container.

9. The method according to claim 8, wherein, The acquisition of model data for the inference model includes: Send an access request to the native code layer to retrieve the model data from the local storage area of ​​the web application through the native code layer.

10. The method according to claim 8, wherein, The weight data is divided into multiple model slices, and the method further includes: In response to the interruption of the download of the multiple model fragments, a new model download request is sent to the server to retrieve the model fragments that have not been downloaded.

11. The method according to claim 1, wherein, The graphics processor is invoked through a target interface. The process of reading weight data from the video memory resources and executing the computational logic of the inference model to obtain the inference result includes: The target interface is detected to determine whether the target interface is available; If the target interface is found to be available, the graphics processor is invoked through the target interface to read the weight data in the video memory resources and execute the calculation logic of the inference model to obtain the inference result.

12. The method according to any one of claims 1-11, wherein, The method further includes: The inference results are displayed in the interactive interface of the web application; the web application runs on the WebView container.

13. A model inference method applied to the native code layer, the method comprising: Get the access request for model data for the inference model sent by the WebView container; The access request is parsed to obtain the original location information; Based on the original location information, the model data of the inference model is read and obtained from the local storage area of ​​the web application; wherein, the web application runs in the WebView container; The model data is provided to the WebView container for inference of inference requests.

14. The method according to claim 13, wherein, The step of reading and obtaining the model data of the inference model from the local storage area of ​​the web application based on the original location information includes: Based on the preset path mapping rules, determine the local location information corresponding to the original location information; Based on the local location information, the model data is read and obtained from the local storage area of ​​the web application.

15. The method according to any one of claims 13-14, wherein, The process of storing the model data in the local storage area includes the following steps: The system receives a data storage request sent by the WebView container and parses the data storage request to obtain the model data, wherein the model data is obtained by the WebView container from the server. The model data in the data storage request is written to the local storage area of ​​the web application, and an index file corresponding to the inference model is generated.

16. A model inference apparatus applied to a WebView container, the apparatus comprising: The acquisition module is used to acquire model data of the inference model in response to an inference request; wherein, the model data includes model structure metadata and weight data; The loading module is used to load the model data into a memory buffer; The processing module is used to create the video memory resources required by the inference model based on the model structure metadata loaded into the memory buffer, and write the weight data loaded into the memory buffer into the video memory resources; The inference module is used to read weight data from the graphics memory resources through the graphics processor and execute the calculation logic of the inference model to obtain the inference result.

17. A model inference apparatus applied at the native code layer, the apparatus comprising: The acquisition module is used to acquire access requests for model data of the inference model sent by the WebView container; The parsing module is used to parse the access request to obtain the original location information; The processing module is used to read and obtain model data of the inference model from the local storage area of ​​the web application based on the original location information; wherein the web application runs in the WebView container; A module is provided for providing the model data to the WebView container for inference of inference requests.

18. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1 to 12, or the method of any one of claims 13 to 15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 12, or the method according to any one of claims 13 to 15.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12, or the method according to any one of claims 13 to 15.

Images