Methods and related apparatus for providing model services on the terminal side
By receiving and executing instructions in the target application to download and run the model, the deployment and use process of large models on the edge is simplified, the problem of high threshold for edge deployment is solved, and efficient and secure model services are achieved.
Patent Information
- Application Number
- CN202510585562.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
In existing technologies, the deployment and use of large-scale models on the edge have a high threshold. Users need to perform complex solution combinations and multiple operations to complete the download and inference of the model.
This paper provides a method for providing model services on the client side. By receiving and executing instructions to download and run the target model in the target application, the user operation process is simplified. This includes receiving instructions to start the application, download the model, and perform inference. Combined with a visual interface and a lightweight authorization mechanism, the deployment and usage threshold is lowered.
It lowers the barrier for users to deploy and use large models on the client side, improves user efficiency, supports rapid integration and invocation of large models on the client side, and enhances security and user experience.
Smart Images

Figure CN123070190A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and more specifically, to a method and related apparatus for providing model services on the client side. Background Technology
[0002] Currently, various large-scale artificial intelligence (AI) models are constantly evolving. By miniaturizing large models and leveraging the built-in computing power, privacy, and free access of personal computers (PCs) and other terminals, large models can be deployed on the terminal side (referred to as the edge side), facilitating their use.
[0003] However, existing edge model deployment tools require users to combine different solutions, find the target combination, download the application tool, and then download the specified model format file via command line or directly from a webpage. During inference, other applications need to be downloaded or inference needs to be completed within the tool itself. Deploying and using large models on the edge presents a significant barrier to entry. Summary of the Invention
[0004] This application provides a method and related apparatus for providing model services on the device side, so as to lower the threshold for deploying and using large models on the device side.
[0005] Firstly, a method for providing model services on the device side is provided. This method can be applied to the device side, such as a terminal or modules within the terminal (e.g., circuits, processors, chips, or chip systems). Taking the application of this method to a terminal as an example, the terminal receives a first instruction to launch a target application, and launches the target application; receives a second instruction to download a target model from the target application, and downloads the target model based on the second instruction; and receives a third instruction to select the target model for inference, and runs the target model for inference based on the third instruction. Using this method, users do not need to worry about model deployment tools; they can download the target model directly from the target application, thereby lowering the barrier to deploying and using large models on the device side. It enables users to quickly use large device-side models on the terminal. It lowers the barrier to users directly using large device-side models and improves the efficiency of user-side usage. Users do not need to worry about model deployment tools; they only need to download the target model from the target application and perform inference directly on the application side. Various lightweight applications that support large device-side models can quickly integrate and call device-side models through this method.
[0006] In one possible implementation, the target application is an application program. Before receiving the first instruction to launch the target application, the method further includes: the terminal receiving a fourth instruction to download the target application, and downloading the target application based on the fourth instruction. Using this method, if the target application is not downloaded on the terminal, the user can instruct the user to download it.
[0007] In another possible implementation, downloading the target model based on the second instruction includes: the terminal downloading the target model in the target application, or selecting and downloading the target model through a visual interface provided by the target application. Using this method, the target model can be downloaded either through the target application or by selecting and downloading it through a visual interface provided by the target application. Users do not need to focus on model deployment tools; they only need to download the target model within the target application, thereby lowering the barrier to deploying and using large models on the device side.
[0008] In another possible implementation, the target application is a browser, and downloading the target model based on the second instruction includes: the terminal selecting and downloading the target model based on a visual interface provided by the browser. Using this method, users do not need to focus on model deployment tools; they only need to download the target model from the target application, thereby lowering the barrier to deploying and using large models on the client side.
[0009] In another possible implementation, the method further includes: the terminal receiving a fifth instruction to configure a plugin in the browser, and configuring the plugin in the browser based on the fifth instruction. Using this method, if a plugin providing a visual interface for model services is not yet configured in the browser before utilizing the browser to provide model services, the user can instruct the plugin to be configured in the browser, for example, by searching for a plugin in the browser and clicking the download and configuration buttons. The terminal, upon receiving the fifth instruction to configure the plugin in the browser, then configures the plugin in the browser based on that fifth instruction.
[0010] In another possible implementation, before downloading the target model based on the second instruction, the method further includes: the terminal sending a first request to the operating system, the first request being used to request authorization information for downloading the target model in the target application; and receiving a first response from the operating system, the first response including the authorization information. This approach, combined with a lightweight local web access authorization mechanism for application permission control, ensures that only authorized applications can access locally detected services, while unauthorized applications cannot access local web services. This improves the security of model services provided on the client side.
[0011] Secondly, an apparatus for providing model services on the endpoint is provided, which can implement the method in the first aspect above. The method can be implemented through software, hardware, or by executing corresponding software through hardware.
[0012] In one possible implementation, the apparatus includes: a receiving unit for receiving a first instruction to launch a target application; a launching unit for launching the target application; the receiving unit further for receiving a second instruction to download a target model in the target application; a downloading unit for downloading the target model based on the second instruction; the receiving unit further for receiving a third instruction to select the target model for inference; and an inference unit for running the target model for inference based on the third instruction.
[0013] Optionally, the target application is an application program; the receiving unit is further configured to receive a fourth instruction to download the target application; and the downloading unit is configured to download the target application based on the fourth instruction.
[0014] Optionally, the download unit is used to download the target model in the target application, or to select and download the target model based on a visual interface provided by the target application.
[0015] Optionally, the target application is a browser; the download unit is used to select the target model and download it based on the visual interface provided by the browser.
[0016] Optionally, the receiving unit is further configured to receive a fifth instruction to configure a plugin in the browser; and the apparatus further includes a configuration unit configured to configure a plugin in the browser based on the fifth instruction.
[0017] Optionally, the apparatus further includes: a sending unit, configured to send a first request to an operating system, the first request being configured to request authorization information for downloading the target model in the target application; and the receiving unit, further configured to receive a first response from the operating system, the first response including the authorization information.
[0018] In another possible implementation, the device includes a processor coupled to a memory; the processor is configured to support the device in performing the corresponding functions described above. The memory is coupled to the processor and stores necessary programs (instructions) and / or data for the device. Optionally, the device may also include an interface for supporting interaction between the device and other devices. Optionally, the memory may be located internally or externally to the device. Optionally, the memory and processor may be integrated.
[0019] Thirdly, a terminal is provided, including an input device, an output device, a memory, and a processor; wherein the processor is coupled to the input device, the output device, and the memory, and the processor is used to execute computer programs or instructions to control the input device to receive information and to control the output device to send information; when the processor executes the computer program or instructions, the processor is also used to implement the above-mentioned method through logic circuits or execution code instructions. The input device and the output device can be input / output interfaces, used to receive signals from other devices outside the terminal and transmit them to the processor, or to send signals from the processor to other devices outside the terminal.
[0020] Wherein, the input device is used to receive a first instruction to launch a target application; the processor is used to launch the target application; the input device is also used to receive a second instruction to download a target model in the target application; the processor is also used to download the target model based on the second instruction; the input device is also used to receive a third instruction to select the target model for inference; and the processor is also used to run the target model for inference based on the third instruction.
[0021] In one possible implementation, the target application is an application program, the input device is further configured to receive a fourth instruction to download the target application, and the processor is further configured to download the target application based on the fourth instruction.
[0022] In another possible implementation, the processor is further configured to download the target model in the target application, or to select and download the target model based on a visual interface provided by the target application.
[0023] In another possible implementation, the target application is a browser, and the processor is further configured to select and download the target model based on a visual interface provided by the browser.
[0024] In yet another possible implementation, the input device is further configured to receive a fifth instruction to configure a plugin in the browser; and the processor is further configured to configure the plugin in the browser based on the fifth instruction.
[0025] In another possible implementation, the output device is configured to send a first request to the operating system, the first request being for requesting authorization information to download the target model in the target application; and the input device is further configured to receive a first response from the operating system, the first response including the authorization information.
[0026] Fourthly, a chip is provided for performing the method described in the first aspect or any implementation thereof.
[0027] Fifthly, a chip module is provided, including an input / output component and a chip, the chip being used to perform the method described in the first aspect or any implementation thereof.
[0028] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program or instructions are stored therein, and when a computer executes the computer program or instructions, the method described in the first aspect or any one of the first aspects is implemented.
[0029] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed on a device, cause the device to perform the method described in the first aspect or any implementation thereof. Attached Figure Description
[0030] Figure 1A A schematic diagram of a neuron structure;
[0031] Figure 1B This is a schematic diagram of the neural network structure;
[0032] Figure 1C This is a schematic diagram of an AI application framework;
[0033] Figure 2 A flowchart illustrating a method for providing model services on the client side, as provided in an embodiment of this application;
[0034] Figure 3 A flowchart illustrating yet another method for providing model services on the client side, as provided in an embodiment of this application;
[0035] Figure 4 A schematic diagram illustrating the provision of model services to the edge.
[0036] Figure 5 A flowchart illustrating yet another method for providing model services on the client side, as provided in an embodiment of this application;
[0037] Figures 6-7 A schematic diagram of the device for providing model services on the end side as provided in the embodiments of this application. Detailed Implementation
[0038] The embodiments of this application are described below with reference to the accompanying drawings.
[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0040] The term "at least one" as used in this application refers to one or more items. "More than one item" means two or more items. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, it should be understood that although the terms "first," "second," etc., may be used in this application to describe various objects, these objects should not be limited to these terms. These terms are only used to distinguish the objects from each other.
[0041] The terms "comprising" and "having," and any variations thereof, used in this application as described below, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or optionally include other steps or units inherent to such processes, methods, products, or apparatus. It should be noted that in this application, words such as "exemplary" or "for example" are used to indicate illustrative, exemplary, or descriptive purposes. Any method or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other methods or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0042] With the rapid development of large model technology, large models have begun to achieve good results in various industries.
[0043] Large models refer to deep learning models with a large or even very large number of parameters, typically containing hundreds of millions, tens of billions, or even trillions of parameters. Large models can also be called foundation models. They are pre-trained on large-scale unlabeled corpora, producing pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include large language models (LLMs) and multi-modal pre-training models. Large language models are a type of artificial intelligence model trained to understand and generate human language. The design and training of large models aim to provide more powerful and accurate model performance to handle more complex and massive datasets or tasks. Large models are typically able to learn more subtle patterns and rules, exhibiting stronger generalization and expressive capabilities.
[0044] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.
[0045] To facilitate understanding, the AI technologies involved in this application will be introduced below. It should be understood that this introduction is not intended to limit this application.
[0046] (1) AI Model
[0047] AI models are the concrete implementations of AI technology functions, representing the mapping relationship between the model's inputs and outputs. AI models can be neural networks, linear regression models, decision tree models, support vector machines (SVMs), Bayesian networks, Q-learning models, or other machine learning (ML) models.
[0048] (2) Neural Network
[0049] Neural networks are a specific implementation of AI or machine learning techniques. According to the general approximation theorem, neural networks can theoretically approximate any continuous function, thus enabling them to learn arbitrary mappings.
[0050] The idea behind neural networks originates from the neuronal structure of the brain. For example, each neuron performs a weighted summation of its input values and outputs the result through an activation function. Figure 1A The diagram shown is a schematic representation of a neuron structure. Assume the neuron's input is x = [x0, x1, ..., x...]. n The weights corresponding to each input are w = [w, w1, ..., w]. n ], where w i As x i The weights are used to assign weights to x. i Weighting is applied. The bias for the weighted sum of the input values is, for example, b. Activation functions can take many forms. Suppose the activation function of a neuron is: y = f(z) = max(0, z), then the output of that neuron is: For example, if the activation function of a neuron is y = f(z) = z, then the output of that neuron is: Among them, b and w i x i The activation function can take various values, such as decimals, integers (e.g., 0, positive integers, or negative integers), or complex numbers. Different neurons in a neural network can have the same or different activation functions.
[0051] Neural networks typically consist of multiple layers, each containing one or more neurons. Increasing the depth and / or width of a neural network enhances its expressive power, providing more robust information extraction and abstract modeling capabilities for complex systems. The depth of a neural network refers to the number of layers it comprises, while the number of neurons in each layer is called its width. In one implementation, the neural network includes an input layer and an output layer. The input layer processes the received input information through neurons and passes the results to the output layer, which then outputs the neural network's output. In another implementation, the neural network includes an input layer, hidden layers, and an output layer, as described in [reference needed]. Figure 1B In a neural network, the input layer processes the received input information through neurons and passes the processing results to the hidden layers. The hidden layers then perform calculations on the received results and pass them to the output layer or an adjacent hidden layer. Finally, the output layer obtains the output of the neural network. A neural network can include one hidden layer or multiple sequentially connected hidden layers; there is no limitation on this.
[0052] Taking neural networks as an example, the AI model involved in this application can be a deep neural network (DNN). Depending on the network construction method, DNN can include feedforward neural networks (FNN), convolutional neural networks (CNN), and recurrent neural networks (RNN).
[0053] (3) Training dataset and inference data
[0054] Training datasets are used to train AI models. A training dataset can include the AI model's input, or it can include both the AI model's input and the target output. Specifically, a training dataset includes one or more training data points, which can be training samples input to the AI model or the AI model's target output. The target output can also be referred to as the label or labeled samples. The training dataset is a crucial part of machine learning; model training essentially involves learning certain features from the training data to make the AI model's output as close as possible to the target output, minimizing the difference between the AI model's output and the target output. The composition and selection of the training dataset can, to a certain extent, determine the performance of the trained AI model.
[0055] Furthermore, a loss function can be defined during the training process of an AI model (such as a neural network). The loss function describes the difference or discrepancy between the output value of the AI model and the target output value. This application does not limit the specific form of the loss function. The training process of an AI model involves adjusting the model parameters of the AI model so that the value of the loss function is less than a threshold, or so that the value of the loss function meets the target requirements. For example, if the AI model is a neural network, adjusting the model parameters of the neural network includes adjusting at least one of the following parameters: the number of layers in the neural network, its width, the weights of the neurons, or the parameters in the activation function of the neurons.
[0056] Inference data can be used as input to a trained AI model for inference. During the model's inference process, the inference data is input into the AI model, and the corresponding output, which is the inference result, is obtained.
[0057] (4) Design of AI Models
[0058] The design of an AI model mainly includes data collection (e.g., collecting training and / or inference data), model training, and model inference. It can also further include the application of the inference results. See also Figure 1CThis illustrates an AI application framework. In the aforementioned data collection phase, the data source provides training and inference data. In the model training phase, the AI model is obtained by analyzing or training the training data provided by the data source. The AI model represents the mapping relationship between the model's input and output. Learning the AI model through model training nodes is equivalent to learning the mapping relationship between the model's input and output using the training data. In the model inference phase, the AI model trained in the model training phase is used to perform inference based on the inference data provided by the data source, yielding the inference result. This phase can also be understood as: inputting inference data into the AI model, obtaining the output through the AI model, which is the inference result. This inference result can indicate the configuration parameters used (executed) by the execution object, and / or the operations performed by the execution object. In the inference result application phase, the inference result is published. For example, the inference result can be uniformly planned by the actor entity, which can send the inference result to one or more execution objects (e.g., core network devices, access network devices, or terminal devices) for execution. For example, the executing entity can also provide feedback on the model's performance to the data source, which facilitates subsequent model updates and training.
[0059] The model parameters can include one or more of the following: model structure parameters (e.g., number of layers, and / or weights), model input parameters (e.g., input dimension, number of input ports), or model output parameters (e.g., output dimension, number of output ports). The input dimension refers to the size of an input data set; for example, if the input data is a sequence, the corresponding input dimension indicates the length of the sequence. The number of input ports refers to the quantity of input data. Similarly, the output dimension refers to the size of an output data set; for example, if the output data is a sequence, the corresponding output dimension indicates the length of the sequence. The number of output ports refers to the quantity of output data.
[0060] AI modules can be used to implement corresponding AI functions. AI modules deployed on different devices can be the same or different. The models of AI modules can achieve different functions depending on the parameter configurations. The models of AI modules can be configured based on one or more of the following parameters: structural parameters (e.g., at least one of the following: number of neural network layers, neural network width, inter-layer connections, neuron weights, neuron activation function, or biases in the activation function), input parameters (e.g., the type and / or dimension of the input parameters), or output parameters (e.g., the type and / or dimension of the output parameters). The biases in the activation function can also be referred to as the biases of the neural network.
[0061] In one example, the neural network mentioned above can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or a generative adversarial network (GAN).
[0062] Deep Neural Networks (DNNs) are artificial neural network architectures with multiple layers of nonlinear transformation units stacked in a hierarchical structure to form deep computational models. Compared to shallow neural networks, deep neural networks have more hidden layers, allowing the network model to capture more complex data structures and higher-level abstract features.
[0063] A CNN is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between a trainable filter and an input image or a convolutional feature map.
[0064] RNN is a type of recursive neural network that takes sequence data as input, recursively moves along the direction of sequence evolution, and connects all nodes (recurrent units) in a chain-like manner.
[0065] GAN is a deep learning model. It consists of a generator and a discriminator, and is trained through adversarial learning. Its purpose is to estimate the potential distribution of data samples and generate new data samples.
[0066] An AI module can have one or more models. A model can infer an output, which includes one or more parameters. The learning, training, or inference processes of different models can be deployed on different nodes or devices, or they can be deployed on the same node or device.
[0067] With the continuous development of the aforementioned large AI models, the use of large models has become more convenient by miniaturizing them and utilizing the data computing power, privacy, and free access of PCs and other terminals to deploy them on the edge.
[0068] However, existing edge model deployment tools require users to combine different solutions, find the target combination, download the application tool, and then download the specified model format file via command line or directly from a webpage. During inference, other applications need to be downloaded or inference needs to be completed within the tool itself. Deploying and using large models on the edge presents a significant barrier to entry.
[0069] Compared to direct API calls from the cloud side, on the client side, users need to go through complex steps to deploy the client-side model and select certain applications or tools before using it. The following are some of the mainstream tools:
[0070] Example of edge deployment tool 1: Users need to download multiple applications to use the graphical model inference experience. This mainly includes the following steps:
[0071] (1) Download the deployment tool for the target.
[0072] You can download the client from a webpage or download the deployment tools for the target using commands.
[0073] (2) Download the model via command line.
[0074] After downloading the deployment tool, you can enter commands in the deployment tool to download the target model.
[0075] (3) Start the target service for download and start the local port service.
[0076] (4) Download and install the graphical application.
[0077] (5) Configure the local service port, select the target local service address and enter it.
[0078] (6) Select the target reasoning model and complete the reasoning.
[0079] Example of edge deployment tool 2: Users need to go through complex steps of downloading and configuring before they can use the edge model for inference. The main steps include:
[0080] (1) Download the deployment tool for the target.
[0081] (2) Download the specified model by applying the search model or by directly accessing and downloading the specified model format via a webpage.
[0082] (3) Configure the model in the deployment tool and store it in the target location.
[0083] For example, after downloading the target model to your local machine, open the download target directory, go to the innermost directory, and paste the downloaded model into it.
[0084] (4) Select the downloaded target reasoning model and complete the reasoning.
[0085] For ordinary users, the above-mentioned edge-side model deployment method has a high barrier to entry, requiring a combination of various applications and steps. Users need to find various tools to run the model on the edge. Users need to understand the basic information of the model, such as its format, to complete the process correctly; otherwise, failures will occur.
[0086] To address this, this application provides a solution for providing model services on the client side. Users do not need to pay attention to model deployment tools and can download the target model in the target application, thereby lowering the barrier to deploying and using large models on the client side.
[0087] like Figure 2 The diagram shown is a flowchart illustrating a method for providing model services on the client side according to an embodiment of this application. Exemplarily, the method may include the following steps:
[0088] S201. Receive the first instruction to start the target application and start the target application.
[0089] When a user needs to use the model for inference, the first step is to open the target application, i.e., send the first command to start the target application. After receiving this first command, the client side starts the target application.
[0090] S202, Receive a second instruction to download the target model in the target application, and download the target model based on the second instruction.
[0091] After launching the target application, the user can search for the target model within the application and instruct it to be downloaded, i.e., send a second instruction to download the target model within the application. Upon receiving this second instruction, the client-side device downloads the target model accordingly.
[0092] S203. Receive the third instruction to select the target model for inference, and run the target model for inference based on the third instruction.
[0093] After downloading the target model, the user can select the target model for inference by sending a third command to select the target model for inference. Upon receiving the third command, the client-side device will run the target model for inference based on the command.
[0094] The above process enables users to quickly access large-scale edge models on the terminal. This lowers the barrier to entry for users to directly use large-scale edge models and improves user efficiency. Users do not need to worry about model deployment tools; they only need to download the target model to the target application and perform inference directly on the application side.
[0095] Various lightweight applications that support large edge models can quickly integrate edge models and call them using this method.
[0096] According to an embodiment of this application, a method for providing model services on the client side allows users to download target models in the target application without needing to pay attention to model deployment tools, thereby lowering the barrier to deploying and using large models on the client side.
[0097] like Figure 3 The diagram shown illustrates another method for providing model services on the client side according to an embodiment of this application. Exemplarily, the method may include the following steps:
[0098] S301. Receive the fourth instruction to download the target application, and download the target application based on the fourth instruction.
[0099] In this embodiment, the target application can be an application (APP). If the target application is not downloaded on the terminal, the user can instruct the user to download it. For example, the user can download the target application through an app store. Specifically, the user can open the app store, search for the target application, and click the download button (i.e., send the fourth instruction to download the target application). The terminal will then download the target application based on this fourth instruction.
[0100] S302: Receive the first instruction to start the target application and start the target application.
[0101] After the target application is downloaded, the user can instruct the terminal to launch it. For example, clicking the target application's icon on the terminal sends the terminal with the first instruction to launch the application. Alternatively, the target application can be launched automatically after downloading.
[0102] S303, Receive the second instruction to download the target model in the target application.
[0103] When a user needs to use the model for inference, they can send a second command to download the target model in the target application, and the terminal will receive this second command. For example, a user can search for the target model in the target application, and after finding the target model, click the download button to send the second command.
[0104] S304, Send the first request to the operating system.
[0105] When downloading the target model, the system first requests authorization from the management service through the operating system's invocation method. That is, it sends a first request to the operating system to request authorization information for downloading the target model in the target application.
[0106] S305: Receive the first response from the operating system.
[0107] After receiving the first request, the operating system invokes the management service to request authorization. Upon receiving the first request, the management service authenticates the target application according to its identity and permissions. If authentication is successful, it starts the detection port, assigns a token, and returns the port and token to the target application. This results in a first response, which includes authorization information. For example, this authorization information includes the port number and token for downloading the target model through the target application.
[0108] Once the target application receives the aforementioned port number and token, if both the port number and the token are valid, the authorization is successful.
[0109] This embodiment combines a lightweight local web access authorization mechanism with application permission control to ensure that only authorized applications can access locally detected services, while unauthorized applications cannot access local web services. This improves the security of the model services provided on the client side.
[0110] S306. Download the target model from the target application, or select and download the target model based on the visual interface provided by the target application.
[0111] After receiving the second instruction to download the target model in the target application, and after obtaining authorization from the management service, the target model can be downloaded either from the target application or by selecting and downloading the target model through the visual interface provided by the target application. Downloading the target model within the target application can be done by entering a command line within the application. Selecting and downloading the target model through the visual interface provided by the target application can be done by searching for the target model within the application's visual interface and then clicking the download button.
[0112] S307. Receive the third instruction to select the target model for inference, and run the target model for inference based on the third instruction.
[0113] After downloading the target model, when the user needs to perform inference using the model, they can select and click on the target model, which sends a third instruction to the terminal to select the target model for inference. Upon receiving this third instruction, the terminal runs the target model for inference based on the instruction. This achieves the ability to select a model and complete inference on the device side.
[0114] For lightweight applications that directly call models for reasoning and question-answering dialogue, this capability can be used to quickly adapt to large models on the calling side for reasoning.
[0115] like Figure 4The diagram illustrates the provision of model services on the client side. APP1 is the target application. The system provides client-side large model services including auto-start service, management service, and model inference service. The vendor provides model invocation services during the large model's runtime. Large models include language models, image models, etc. When a user launches APP1, searches for the target model within APP1. APP1 requests authorization to download the target model from the management service. The management service sends the authorization information, and the user downloads the searched target model from APP1. The system's auto-start service then initiates the target model for inference. By providing a secure access scheme with two layers of services (auto-start service and management service), the barrier to entry for users to directly use client-side large models is lowered, improving user efficiency on the client side.
[0116] According to an embodiment of this application, a method for providing model services on the device side allows users to avoid the need to focus on model deployment tools. They only need to download the target application, and then download the target model from the target application or the visual interface provided by the target application. Inference can then be performed directly in the target application, thereby reducing the threshold for deploying and using large models on the device side.
[0117] like Figure 5 As shown, another method for providing model services on the client side is provided in an embodiment of this application. Exemplarily, the method may include the following steps:
[0118] S501: Receive the fifth instruction to configure the plugin in the browser, and configure the plugin in the browser based on the fifth instruction.
[0119] This embodiment describes a scheme for providing model services using a browser on the client side. Before using the browser to provide model services, if the browser does not yet have a plugin configured to provide a visual interface for model services, the user can instruct the browser to configure the plugin, for example, by searching for a plugin in the browser and clicking the download and configuration buttons. Upon receiving the fifth instruction to configure the plugin in the browser, the client then configures the plugin in the browser based on that instruction.
[0120] S502: Receive the first instruction to open the browser and launch the browser.
[0121] For example, when a user opens a browser, the first instruction to open the browser is sent. The terminal receives this first instruction and opens the browser.
[0122] S503: Receive the second instruction to download the target model in the browser.
[0123] For example, in the visual interface provided by the browser, there is a download button for the target model. When a user clicks the download button, it sends a second command to download the target model in the browser.
[0124] S504: Send the first request to the operating system.
[0125] When downloading the target model, the system first requests authorization from the management service through the operating system's invocation method. That is, it sends a first request to the operating system to request authorization information for downloading the target model in the target application.
[0126] S505 receives a first response from the operating system, which includes authorization information.
[0127] After receiving the first request, the operating system invokes the management service to request authorization. Upon receiving the first request, the management service authenticates the target application according to its identity and permissions. If authentication is successful, it starts the detection port, assigns a token, and returns the port and token to the target application. This results in a first response, which includes authorization information. For example, this authorization information includes the port number and token for downloading the target model through the target application.
[0128] Once the target application receives the aforementioned port number and token, if both the port number and the token are valid, the authorization is successful.
[0129] This embodiment combines a lightweight local web access authorization mechanism with application permission control to ensure that only authorized applications can access locally detected services, while unauthorized applications cannot access local web services. This improves the security of the model services provided on the client side.
[0130] S506. Select and download the target model using the visual interface provided by the browser.
[0131] After receiving the second instruction to download the target model in the browser, and after obtaining authorization from the management service, the target model can be selected and downloaded using the visual interface provided by the browser. Selecting and downloading the target model using the browser's visual interface can be done by searching for the target model in the browser's visual interface and then clicking the download button.
[0132] S507. Receive the third instruction to select a target model for inference, and run the target model for inference based on the third instruction.
[0133] Various lightweight applications that support large client-side models can quickly integrate and call client-side models in this way, while also being compatible with existing browser-based web pages and plugins.
[0134] This capability can also be reused for web pages and plugins that have already been adapted to the existing services of the large-scale client-side model in the ecosystem, and a secure access mechanism is adopted to ensure secure access by general web browsers.
[0135] For application scenarios with high security and privacy requirements, this capability can be directly used to access edge capabilities to address the actual needs of users without network access and for privacy, thereby rapidly improving productivity in user privacy scenarios.
[0136] According to an embodiment of this application, a method for providing model services on the device side allows users to download the target model by configuring a plugin in the browser and select the target model on the device side to complete inference without having to pay attention to the model deployment tool. This reduces the barrier to deploying and using large models on the device side.
[0137] Based on the same concept as the above methods, embodiments of this application also provide an apparatus for providing model services on the terminal side, which is used to implement the various methods described above. This apparatus for providing model services on the terminal side can be a terminal as described in the above method embodiments, or a circuit or chip within the terminal. It is understood that, in order to implement the above functions, this apparatus for providing model services on the terminal side includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0138] This application embodiment can divide the device providing model services on the terminal side into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing unit. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.
[0139] Based on the same concept as the above-described method for providing model services on the client side, this application also provides the following apparatus for providing model services on the client side:
[0140] like Figure 6 The diagram shown is a schematic representation of a device for providing model services on the edge, according to an embodiment of this application. The device 6000 for providing model services on the edge includes:
[0141] The receiving unit 601 is configured to receive a first instruction to launch a target application; the launching unit 602 is configured to launch the target application; the receiving unit 601 is further configured to receive a second instruction to download a target model in the target application; the downloading unit 603 is configured to download the target model based on the second instruction; the receiving unit 601 is further configured to receive a third instruction to select the target model for inference; and the inference unit 604 is configured to run the target model for inference based on the third instruction.
[0142] Optionally, the target application is an application program; the receiving unit 601 is further configured to receive a fourth instruction to download the target application; and the downloading unit 603 is configured to download the target application based on the fourth instruction.
[0143] Optionally, the download unit 603 is used to download the target model in the target application, or to select and download the target model based on the visual interface provided by the target application.
[0144] Optionally, the target application is a browser; the download unit 603 is used to select the target model and download it based on the visual interface provided by the browser.
[0145] Optionally, the receiving unit 601 is further configured to receive a fifth instruction for configuring a plugin in the browser; and the device 6000 further includes ( Figure 6 (Not shown in the image): Configuration unit, used to configure the plugin in the browser based on the fifth instruction.
[0146] Optionally, the device 6000 further includes ( Figure 6 (Not shown in the image): The sending unit is configured to send a first request to the operating system, the first request being used to request authorization information for downloading the target model in the target application; and the receiving unit 601 is further configured to receive a first response from the operating system, the first response including the authorization information.
[0147] For details regarding the specific implementation of the receiving unit 601, the starting unit 602, the downloading unit 603, the inference unit 604, the configuration unit, and the sending unit, please refer to the description in the above method embodiments.
[0148] According to an embodiment of this application, an apparatus for providing model services on the device side allows users to download target models in target applications without needing to pay attention to model deployment tools, thereby lowering the barrier to deploying and using large models on the device side.
[0149] like Figure 7 The diagram shown is a structural schematic of another device for providing model services on the end side according to an embodiment of this application.
[0150] The device 7000 for providing services includes one or more processors 704. Figure 7 An example of a processor is shown in the diagram. The device 7000 providing model services on the edge may further include an input device 701 and an output device 702, with the processor 704 coupled to the input device 701 and the output device 702. The device 7000 providing model services on the edge may also include a memory 703. The memory 703 is used to store instructions executed by the processor 704, or to store input data required by the processor 703 to execute instructions, or to store data generated after the processor 704 executes instructions.
[0151] Wherein, the input device 701 is used to receive a first instruction to start a target application; the processor 704 is used to start the target application; the input device 701 is also used to receive a second instruction to download a target model in the target application; the processor 704 is also used to download the target model based on the second instruction; the input device 701 is also used to receive a third instruction to select the target model for inference; and the processor 704 is also used to run the target model for inference based on the third instruction.
[0152] In one possible implementation, the target application is an application program, the input device 701 is further configured to receive a fourth instruction to download the target application; and the processor 704 is further configured to download the target application based on the fourth instruction.
[0153] In another possible implementation, the processor 704 is further configured to download the target model in the target application, or to select and download the target model based on a visual interface provided by the target application.
[0154] In another possible implementation, the target application is a browser, and the processor 704 is further configured to select and download the target model based on a visual interface provided by the browser.
[0155] In yet another possible implementation, the input device 701 is further configured to receive a fifth instruction to configure a plugin in the browser; and the processor 704 is further configured to configure a plugin in the browser based on the fifth instruction.
[0156] In another possible implementation, the output device 702 is configured to send a first request to the operating system, the first request being for requesting authorization information to download the target model in the target application; and the input device 701 is further configured to receive a first response from the operating system, the first response including the authorization information.
[0157] According to an embodiment of this application, an apparatus for providing model services on the device side allows users to download target models in target applications without needing to pay attention to model deployment tools, thereby lowering the barrier to deploying and using large models on the device side.
[0158] The module division in this application is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in the various examples of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0159] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.
[0160] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the methods described in the above embodiments.
[0161] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the methods described in the above embodiments.
[0162] This application also provides a circuit coupled to a memory, which is used to perform the methods shown in the above embodiments. This circuit may include a chip circuit.
[0163] This application also provides a chip for performing the methods shown in the above embodiments.
[0164] This application also provides a chip module, including an input / output component and a chip, wherein the chip is used to perform the methods shown in the above embodiments.
[0165] It should be noted that one or more of the above units can be implemented by software, hardware, or a combination of both. When any of the above units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow.
[0166] In this application, the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or all or part of the circuitry in the aforementioned devices used to implement the processing functions, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in this application can be directly embodied in the execution of the hardware processor, or can be executed by a combination of hardware and software modules within the processor.
[0167] When the above units or components are implemented in hardware, the hardware can be any one or any combination of a CPU, microprocessor, digital signal processing (DSP) chip, microcontroller unit (MCU), artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.
[0168] Optionally, embodiments of this application also provide a chip system, including: at least one processor and an interface, wherein the at least one processor is coupled to a memory via the interface, and when the at least one processor executes a computer program or instructions in the memory, the chip system performs the method in any of the above method embodiments. Optionally, the chip system may be composed of chips, or may include chips and other discrete devices; embodiments of this application do not specifically limit this.
[0169] The memory in this application can also be a circuit or any other device capable of performing storage functions, used to store program instructions and / or data. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. For example, memory can be non-volatile memory, such as digital versatile disc (DVD), hard disk drive (HDD), or solid-state drive (SSD), or it can be volatile memory, such as random-access memory (RAM).
[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0171] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0172] It is understood that the term "embodiment" used throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, various embodiments throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.
[0173] It should be understood that in the description of this application, unless otherwise stated, " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B can represent A or B; where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply difference. In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner to facilitate understanding.
[0174] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, network device, or data center to another website, computer, network device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0175] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, a single processor or other unit may implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0176] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.
[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0178] The components in the device described in this application embodiment can be combined, divided, or removed according to actual needs. Those skilled in the art can combine or integrate the different embodiments and features described in this specification.
[0179] In this application, examples may reference each other without logical contradiction. For example, methods and / or terms between method embodiments may reference each other, functions and / or terms between device embodiments may reference each other, and functions and / or terms between device examples and method examples may reference each other.
Claims
1. A method for providing model services on the client side, characterized in that, The method includes: Receive a first instruction to launch the target application, and launch the target application; Receive a second instruction to download a target model in the target application, and download the target model based on the second instruction; Receive a third instruction to select the target model for inference, and run the target model for inference based on the third instruction.
2. The method as described in claim 1, characterized in that, The target application is an application program, and before receiving the first instruction to launch the target application, the method further includes: Receive a fourth instruction to download the target application, and download the target application based on the fourth instruction.
3. The method as described in claim 2, characterized in that, The downloading of the target model based on the second instruction includes: Download the target model from the target application, or select and download the target model based on the visual interface provided by the target application.
4. The method as described in claim 1, characterized in that, The target application is a browser, and the downloading of the target model based on the second instruction includes: Select the target model and download it using the visual interface provided by the browser.
5. The method as described in claim 4, characterized in that, The method further includes: Receive a fifth instruction to configure the plugin in the browser, and configure the plugin in the browser based on the fifth instruction.
6. The method according to any one of claims 1-5, characterized in that, Before downloading the target model based on the second instruction, the method further includes: Send a first request to the operating system, the first request being used to request authorization information for downloading the target model in the target application; Receive a first response from the operating system, the first response including the authorization information.
7. An apparatus for providing model services on the end side, characterized in that, The device includes: The receiving unit is used to receive the first instruction to start the target application; A startup unit is used to start the target application; The receiving unit is further configured to receive a second instruction to download the target model in the target application; Download unit, used to download the target model based on the second instruction; The receiving unit is also configured to receive a third instruction to select the target model for inference; The inference unit is used to run the target model for inference based on the third instruction.
8. The apparatus as claimed in claim 7, characterized in that, The target application is an application program; The receiving unit is also configured to receive a fourth instruction to download the target application; The download unit is used to download the target application based on the fourth instruction.
9. The apparatus as claimed in claim 8, characterized in that, The download unit is used to download the target model from the target application, or to select and download the target model based on the visual interface provided by the target application.
10. The apparatus as claimed in claim 7, characterized in that, The target application is a browser; The download unit is used to select the target model and download it based on the visual interface provided by the browser.
11. The apparatus as claimed in claim 10, characterized in that, The receiving unit is also configured to receive a fifth instruction for configuring the plugin in the browser; The device further includes: The configuration unit is used to configure the plugin in the browser based on the fifth instruction.
12. The apparatus according to any one of claims 7-11, characterized in that, The device further includes: The sending unit is configured to send a first request to the operating system, wherein the first request is used to request authorization information for downloading the target model in the target application; The receiving unit is further configured to receive a first response from the operating system, the first response including the authorization information.
13. A terminal, characterized in that, It includes an input device, an output device, a memory, and a processor; wherein the processor is used to perform the method as described in any one of claims 1-6.
14. A chip, characterized in that, The chip is used to perform the method as described in any one of claims 1-6.
15. A chip module, characterized in that, It includes input / output components and a chip, said chip being used to perform the method as described in any one of claims 1-6.
16. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a device providing model services on the terminal side, implement the method as described in any one of claims 1-6.
