Method, device and equipment for model reasoning, storage medium and program product

Abstract

According to the embodiment of the invention, a method and device for model reasoning, equipment, a storage medium and a program product are provided. The method comprises: at a client, in response to receiving a model input, determining a feature representation of the model input by using a first machine learning model locally deployed by the client, the first machine learning model being a part of a second machine learning model; the feature representation and prompt information are at least sent to the server side so that the feature representation can be continuously processed through the second machine learning model, and the prompt information at least indicates the proportion of the first machine learning model in the second machine learning model; and obtaining a model output corresponding to the model input from the server, wherein the model output is generated by processing the feature representation by the second machine learning model.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, electronic devices, computer-readable storage media, and computer program products for model reasoning. Background Technology

[0002] With the development of machine learning technology, machine learning models can now be used to perform tasks in various application environments. Because inference tasks in machine learning models require significant computational and storage space, these models are currently typically deployed in the cloud. User clients can upload model inputs to the cloud to use the cloud-based machine learning model for inference. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for model inference is provided. The method is implemented on a client and includes: in response to receiving model input, determining a feature representation of the model input using a first machine learning model deployed locally on the client, wherein the first machine learning model is part of a second machine learning model; sending at least the feature representation and prompt information to a server to further process the feature representation using the second machine learning model, the prompt information indicating at least the proportion of the first machine learning model in the second machine learning model; and obtaining a model output corresponding to the model input from the server, the model output being generated by the second machine learning model through processing the feature representation.

[0004] In a second aspect of this disclosure, a method for model inference is provided. The method is implemented on a server and includes: receiving from a client at least a feature representation and prompt information, the feature representation being obtained by processing model inputs using a first machine learning model deployed locally on the client, the prompt information indicating at least the proportion of the first machine learning model in a second machine learning model; processing the feature representation based on the prompt information using the second machine learning model to determine a model output corresponding to the model input; and sending the model output to the client.

[0005] In a third aspect of this disclosure, an apparatus for model inference is provided. The apparatus is implemented on a client and includes: a feature representation determination module configured to, in response to receiving a model input, determine a feature representation of the model input using a first machine learning model deployed locally on the client, wherein the first machine learning model is part of a second machine learning model; a feature representation sending module configured to send at least the feature representation and prompt information to a server to further process the feature representation using the second machine learning model, wherein the prompt information at least indicates the proportion of the first machine learning model in the second machine learning model; and a model output acquisition module configured to acquire a model output corresponding to the model input from the server, the model output being generated by the second machine learning model through processing the feature representation.

[0006] In a fourth aspect of this disclosure, an apparatus for model inference is provided. The apparatus is implemented on a server side and includes: a feature representation receiving module configured to receive at least a feature representation and prompt information from a client, wherein the feature representation is obtained by processing a model input using a first machine learning model deployed locally on the client, and the prompt information indicates at least the proportion of the first machine learning model in a second machine learning model; a feature representation processing module configured to process the feature representation based on the prompt information using the second machine learning model to determine a model output corresponding to the model input; and a model output sending module configured to send the model output to the client.

[0007] In a fifth aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform a method according to a first or second aspect of this disclosure when executed by the at least one processor.

[0008] In a sixth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which, when executed by a processor, cause the processor to perform a method according to a first or second aspect of this disclosure.

[0009] In a seventh aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first or second aspect.

[0010] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] In the following detailed description, in conjunction with the accompanying drawings, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown; Figure 2A An example flow of model reasoning according to some embodiments of this disclosure is shown; Figure 2B Example signaling flows for model inference according to some embodiments of this disclosure are shown; Figure 2C An example of a second machine learning model according to some embodiments of this disclosure is shown; Figure 2D Examples of model reasoning according to some embodiments of this disclosure are shown; Figure 3 A flowchart of a method for model inference according to some embodiments of the present disclosure is shown; Figure 4 A flowchart of a method for model inference according to other embodiments of the present disclosure is shown; Figure 5 An exemplary structural block diagram of an apparatus for model inference according to some embodiments of the present disclosure is shown; Figure 6 Exemplary structural block diagrams of an apparatus for model inference according to other embodiments of the present disclosure are shown; and Figure 7 A block diagram of an electronic device that can implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0015] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0018] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0021] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0022] For ease of understanding, some of the terms used in the embodiments of this disclosure will be explained below.

[0023] Attention: A core computational model that mimics the human ability to selectively focus on information. By dynamically allocating weights, it allows the model to focus on key information when processing sequential data.

[0024] Vectorization (Embedding): Converting a token into a multi-dimensional vector.

[0025] Model service: The model is deployed in the cloud and provides services to the outside world through standardized interfaces. The interface can be called through a transmission protocol (such as Hypertext Transfer Protocol (HTTP)) to use the model for inference.

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, an application 115 is installed on a client 110. A user 140 can interact with the application 115 via the client 110 and / or an attached device of the client 110. For example, the application 115 can capture the user 140's voice via an audio capture device (e.g., a microphone) of the client 110, can capture the user 140's image via an image capture device (e.g., a camera) of the client 110, and so on.

[0027] In embodiments of this disclosure, application 115 can be any suitable application such as a browser application, social networking application, or media item application. In environment 100, if application 115 is active, client 110 can present page 150 of application 115. Page 150 can include various types of pages that application 115 can provide. In some embodiments, client 110 and / or application 115 can receive user interaction operations via page 150.

[0028] In some embodiments, a communication connection is established between the client 110 and the server 120. The communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections, and the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the client 110 and the server 120 can perform signaling interaction through their communication connection to provide services to the application 115.

[0029] like Figure 1 As shown, server 120 can invoke machine learning model 130 to provide services to application 115 based on the output of machine learning model 130. Machine learning model 130 can be deployed on server 120 or on other devices. Machine learning model 130 can be based on any suitable model architecture, including but not limited to Transformer models, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep neural networks (DNNs), etc. In some embodiments, machine learning model 130 can be based on language models (LMs), large language models (LLMs), etc. Language models, by learning from large amounts of corpora, are capable of question answering. Machine learning model 130 can also be based on other suitable models.

[0030] Client 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0031] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 120 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0032] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0033] As mentioned earlier, a user's client can upload model input to the cloud to use a cloud-based machine learning model for inference. If the model input includes the user's private data (such as text, images, and videos), the cloud can employ methods like confidential computing to ensure the model service runs in a secure environment. This leverages computer hardware capabilities to encrypt data in memory and on disk, guaranteeing the security of the entire computation process, particularly data and computation during inference, thus preventing data leakage from a technical perspective. However, this only ensures data security during the model's computation process; it doesn't guarantee data security during the client's data upload to the server. The data upload process carries the risk of data leaving the domain, potentially leading to privacy data leaks.

[0034] In view of this, this disclosure proposes an improved scheme for model inference. According to an embodiment of this disclosure, a first machine learning model deployed locally on the client can be used to determine the feature representation of the model input, wherein the first machine learning model is part of a second machine learning model. The client sends at least the feature representation and prompt information to the server to continue processing the feature representation using the second machine learning model, the prompt information indicating at least the proportion of the first machine learning model in the second machine learning model. The server can receive at least the feature representation and prompt information from the client and, using the second machine learning model, processes the feature representation based on the prompt information to determine the model output corresponding to the model input. The server sends the model output to the client. The client obtains the model output corresponding to the model input from the server.

[0035] In this way, the first machine learning model is used to process the model input at the client end. The client provides the server with the feature representation output by the first machine learning model instead of the original model input. This can encrypt the model input and reduce the risk of privacy leakage caused by data going out of the domain.

[0036] The exemplary embodiments of this disclosure will be discussed in detail below with reference to the accompanying drawings.

[0037] First refer to Figure 2A , Figure 2A An example flow 200A for model reasoning according to some embodiments of this disclosure is shown. For example... Figure 2A As shown, after receiving the model input 201, the client 110 can sequentially perform vectorization processing (202), attention calculation (203), and encoding calculation (204) on the model input 201.

[0038] Client 110 can process model input 201 into a continuous, learnable low-dimensional vector by performing vectorization processing on the model input 201. Client 110 can then perform attention calculations on the vector to obtain attention-weighted vectors. Client 110 can further encode the attention-weighted vectors (e.g., nonlinear transformation (FFN), residuals, etc.) to obtain the feature representations that will be provided to downstream tasks / next layers.

[0039] Client 110 can provide feature representations to server 120. In embodiments of this disclosure, client 110 may deploy a first machine learning model 202, which can be used to perform vectorization, attention calculation, and encoding calculation on model input 201. The first machine learning model 202 may be part of a second machine learning model 207 deployed on server 120. The second machine learning model 207 may, for example, correspond to... Figure 1 The machine learning model 130 is shown in the figure. The second machine learning model 207 can be based on any suitable model structure. For example, the second machine learning model 207 can be based on a Transformer structure, a Transformer Decode-only structure, etc.

[0040] The second machine learning model 207 can be a general model for various scenarios or a specific model for a particular scenario. The first machine learning model 202 and the second machine learning model 207 can be configured to perform any appropriate task as needed, including question answering, recommendation, content generation (e.g., text-to-image, image-to-image, text-to-audio or video, etc.), object recognition, conversion, etc. The first machine learning model 202 deployed at client 110 can be part of the trained second machine learning model 207. In practice, the second machine learning model 202 can be trained using an appropriate training dataset based on the scenario or task category that the first and second machine learning models 202 are targeting. If the second machine learning model 207 is a general model, it can be trained using a training dataset corresponding to a general scenario. If the second machine learning model 207 is a specific model, it can be trained using a training dataset corresponding to a specific scenario. If the second machine learning model 207 is used to perform a specific type of task, it can be trained using a training dataset corresponding to that specific type of task.

[0041] It should be noted that the first machine learning model 202 is a part of the second machine learning model 207 starting from the first layer (e.g., input layer, embedding layer, etc.). For example, if the second machine learning model 207 includes a 10-layer network structure, the first machine learning model 202 can be a continuous network structure that includes at least the first layer (e.g., only the first layer network structure, the first two layers network structure, the first three layers network structure, etc.).

[0042] To facilitate the second machine learning model 207's understanding of the proportion of the first machine learning model 202 within the second machine learning model 207, the client 110 can also provide prompt information, which can at least indicate the proportion of the first machine learning model 202 in the second machine learning model 207. The server 120 can, in response to receiving the feature representation, perform model service (207) on the feature representation. The model service can invoke the second machine learning model 207 to process the feature representation based on the prompt information. The second machine learning model 207 can determine the model output 208, which is also the output corresponding to model output 201.

[0043] The server 120 can, for example, determine the proportion of the first machine learning model 202 in the second machine learning model 207 based on the prompt information, and according to this proportion, continue processing the feature representation using the portion of the second machine learning model 207 other than the first machine learning model 202 to determine the model output 206. For example, if the second machine learning model 207 includes a 10-layer network structure, and the first machine learning model 202 is the first layer of the second machine learning model 207, then the second machine learning model 207 can continue processing the feature representation starting from the second layer of the network structure according to the prompt information to determine the model output 208.

[0044] Figure 2B An example signaling flow 200B based on model reasoning according to some embodiments of this disclosure is shown. The example signaling flow 200B relates to client 110 and server 120. (As...) Figure 2B As shown, client 110 can determine (211) the model input. In some embodiments, client 110 can receive user input from user 140 via page 150 and determine the model input based on the user input. User input may include text, images, videos, etc. In some embodiments, client 110 may also determine the model input directly based on local data. This disclosure does not limit the specific method of determining the model input.

[0045] In response to receiving model input, client 110 may determine (212) the feature representation of the model input using a first machine learning model (e.g., first machine learning model 202) deployed locally on the client, wherein the first machine learning model is part of a second machine learning model (e.g., second machine learning model 207).

[0046] In some embodiments, the second machine learning model includes a model structure of an encoder and a decoder (e.g., the second machine learning model is a machine learning model based on a Transformer architecture), the encoder includes multiple encoder layers, and the first machine learning model may include at least one encoder layer from the multiple encoder layers. It is understood that this at least one encoder layer may be at least one consecutive encoder layer that includes at least a first encoder layer.

[0047] In other embodiments, the second machine learning model includes a decoder-only model structure (e.g., the second machine learning model is a machine learning model based on a Transformer Decode-only architecture), the decoder includes multiple decoder layers, and the first machine learning model includes at least one of the multiple decoder layers. It is understood that this at least one decoder layer can be at least one consecutive decoder layer that includes at least the first decoder layer.

[0048] refer to Figure 2C , Figure 2C Example 200C of a second machine learning model based on a Transformer Decode-only architecture according to some embodiments of the present disclosure is shown. The second machine learning model may include an embedding layer 221, a decoder 222, and a sampler 226. The decoder 222 may also include N decoder layers (N being a positive integer), each decoder layer may include a normalization layer 223, a self-attention layer 224, and a multilayer perceptron (MLP) 225. Reference Figure 2D , Figure 2D Example 200D of model inference according to some embodiments of the present disclosure is shown. If the second machine learning model 207 on the server side 120 is the machine learning model shown in example 200C, the first machine learning model 202 on the client side 110 may include the embedding layer 221 and the first decoder layer 222-1 of the second machine learning model 207.

[0049] In some embodiments, the first machine learning model deployed on different clients can be different. Before deploying the first machine learning model, each client 110 can determine its own local computing power. Client 110 can determine its local computing power, for example, based on its own processor (e.g., CPU) performance, memory, device version, etc. Client 110 can determine the proportion of the first machine learning model to be deployed on the client relative to the second machine learning model based on its local computing power, and deploy the first machine learning model on client 110 based on the proportion. Thus, the proportion of the first machine learning model deployed locally on clients with better computing power can be higher than the proportion of the first machine learning model deployed locally on clients with weaker computing power. The stronger the local computing power of a client, the larger the scale of the deployed first machine learning model, and the stronger the computing power.

[0050] Return to reference Figure 2BClient 110 may send at least (213) feature representation and prompt information to server 120 to continue processing the feature representation using the second machine learning model. The prompt information indicates at least the proportion of the first machine learning model in the second machine learning model. In some embodiments, server 120 may provide services to different clients. Client 110 may also send a client identifier to server 120 so that server 120 may subsequently return the model output to the client 110 corresponding to the client identifier. Of course, client 110 may also send other information to server 120, such as context information of the model input. This disclosure does not limit the specific other information.

[0051] Server 120 can receive at least (214) feature representation and prompt information from client 110. It should be noted that, in some embodiments, to save transmission resources, improve transmission efficiency, and enhance transmission security, client 110 can also perform data encoding operations on the feature representation and prompt information to obtain encoded information. The data encoding operations include encryption and / or compression operations. Client 110 can send the encoded information to server 120. Server 120 can receive the encoded information and perform data decoding operations on it to obtain at least the feature representation and prompt information. The data decoding operations include decryption and / or decompression operations. Thus, feature representation can be encoded, further enhancing data security.

[0052] In response to the acquisition of feature representations and prompts, server 120 can utilize the second machine learning model to process the feature representations based on the prompts in order to determine the model output corresponding to the model input (215). Server 120 can deploy a complete second machine learning model, or it can deploy only the part of the second machine learning model other than the first machine learning model (this part can be referred to as the third machine learning model). For example, server 120 can deploy the complete second machine learning model shown in example 200C, or it can deploy the third machine learning model 231 shown in 200C, which only includes the second decoder layer 222-2 to the Nth decoder layer 222-N (that is, a total of N-1 decoder layers) and sampler 226 in the N-layer decoder layer.

[0053] In some embodiments, server 120 may determine the prompt word input for a second or third machine learning model based at least on feature representation and prompt information, and process the feature representation by providing the prompt word input to the second or third machine learning model. After determining the model output, server 120 may send (216) the model output to client 110. Client 110 may receive the model output accordingly. In some embodiments, client 110 may provide the model output to user 140 via page 150.

[0054] Therefore, by using a first machine learning model to process the model input at the client end, and providing the server with the feature representation output by the first machine learning model instead of the original model input, encryption of the model input can be achieved, reducing the risk of privacy leaks due to data going out of domain. Furthermore, a second machine learning model deployed on the server can be used to further process the feature representation to determine the final feature output. In the autoregressive iterative output, the data flow no longer bypasses the client side, achieving a good balance between performance and security.

[0055] Figure 3 A flowchart of a method 300 for model inference according to some embodiments of the present disclosure is shown. Method 300 can be implemented in... Figure 1 The client is at location 110. (Refer to...) Figure 1 The environment 100 is used to describe method 300.

[0056] In box 310, in response to receiving model input, client 110 determines the feature representation of the model input using a first machine learning model deployed locally on the client, wherein the first machine learning model is part of a second machine learning model.

[0057] In box 320, client 110 sends at least feature representations and prompts to server to continue processing feature representations using the second machine learning model, the prompts indicating at least the proportion of the first machine learning model in the second machine learning model.

[0058] In box 330, client 110 obtains the model output corresponding to the model input from the server. The model output is generated by the second machine learning model by processing the feature representation.

[0059] In some embodiments, method 300 further includes: determining the local computing power of the client; determining the proportion of the first machine learning model to be deployed to the client in the second machine learning model based on the local computing power; and deploying the first machine learning model on the client based on the proportion.

[0060] In some embodiments, sending at least the feature representation and prompt information to the server includes: performing a data encoding operation on at least the feature representation and prompt information to obtain encoded information, the data encoding operation including encryption and / or compression operations; and sending the encoded information to the server.

[0061] In some embodiments, the second machine learning model includes a model structure of an encoder and a decoder, the encoder including multiple encoder layers, and the first machine learning model includes at least one encoder layer among the multiple encoder layers.

[0062] In some embodiments, the second machine learning model includes a decoder-only model structure, the decoder including multiple decoder layers, and wherein the first machine learning model includes at least one of the multiple decoder layers.

[0063] Figure 4 A flowchart of a method 400 for model inference according to other embodiments of the present disclosure is shown. Method 400 can be implemented in... Figure 1 There are 120 server-side locations. (Refer to...) Figure 1 The environment 100 is used to describe method 400.

[0064] In box 410, server 120 receives at least feature representation and prompt information from client. Feature representation is obtained by processing model input by a first machine learning model deployed locally on client. Prompt information indicates at least the proportion of the first machine learning model in the second machine learning model.

[0065] In box 420, server 120 uses a second machine learning model to process feature representations based on prompts in order to determine the model output corresponding to the model input.

[0066] In box 430, server 120 sends model output to client.

[0067] In some embodiments, the server 120 deploys a second machine learning model or a third machine learning model, wherein the third machine learning model is the part of the second machine learning model other than the first machine learning model.

[0068] In some embodiments, using a second machine learning model to process feature representations based on cue information includes: determining cue word input for the second or third machine learning model based at least on the feature representations and cue information; and providing the cue word input to the second or third machine learning model to process the feature representations using the second or third machine learning model.

[0069] In some embodiments, receiving at least feature representation and prompt information from the client includes: receiving encoded information from the client, the encoded information being obtained by performing data encoding operations on at least the feature representation and prompt information, the data encoding operations including encryption operations and / or compression operations; and performing data decoding operations on the encoded information to obtain at least the feature representation and prompt information, the data decoding operations including decryption operations and / or decompression operations.

[0070] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 5 An exemplary structural block diagram of an apparatus 500 for model inference according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in a client 110. Various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0071] like Figure 5 As shown, the apparatus 500 includes a feature representation determination module 510, configured to determine the feature representation of the model input using a first machine learning model deployed locally on the client, in response to receiving model input, wherein the first machine learning model is part of a second machine learning model. The apparatus 500 also includes a feature representation sending module 520, configured to send at least the feature representation and prompt information to a server to further process the feature representation using the second machine learning model, wherein the prompt information at least indicates the proportion of the first machine learning model in the second machine learning model. The apparatus 500 also includes a model output acquisition module 530, configured to acquire the model output corresponding to the model input from the server, wherein the model output is generated by the second machine learning model through processing the feature representation.

[0072] In some embodiments, the apparatus 500 further includes: a capability determination module configured to determine the local computing power of the client; a proportion determination module configured to determine the proportion of the first machine learning model to be deployed to the client in the second machine learning model based on the local computing power; and a model deployment module configured to deploy the first machine learning model on the client based on the proportion.

[0073] In some embodiments, the feature representation sending module 520 is further configured to: perform data encoding operations on at least the feature representation and the prompt information to obtain encoded information, the data encoding operations including encryption operations and / or compression operations; and send the encoded information to the server.

[0074] In some embodiments, the second machine learning model includes a model structure of an encoder and a decoder, the encoder including multiple encoder layers, and the first machine learning model includes at least one encoder layer among the multiple encoder layers.

[0075] In some embodiments, the second machine learning model includes a decoder-only model structure, the decoder including multiple decoder layers, and wherein the first machine learning model includes at least one of the multiple decoder layers.

[0076] Figure 6 An exemplary structural block diagram of an apparatus 600 for model inference according to other embodiments of the present disclosure is shown. Apparatus 600 may be implemented as or included in server 120. Various modules / components in apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] like Figure 6 As shown, the device 600 includes a feature representation receiving module 610, configured to receive at least a feature representation and prompt information from a client. The feature representation is obtained by processing the model input by a first machine learning model deployed locally on the client. The prompt information indicates at least the proportion of the first machine learning model in a second machine learning model. The device 600 also includes a feature representation processing module 620, configured to process the feature representation based on the prompt information using the second machine learning model to determine the model output corresponding to the model input. The device 600 also includes a model output sending module 630, configured to send the model output to the client.

[0078] In some embodiments, the server 120 deploys a second machine learning model or a third machine learning model, wherein the third machine learning model is the part of the second machine learning model other than the first machine learning model.

[0079] In some embodiments, the feature representation processing module 620 is further configured to: determine a prompt word input for a second machine learning model or a third machine learning model based at least on the feature representation and prompt information; and provide the prompt word input to the second machine learning model or the third machine learning model to process the feature representation using the second machine learning model or the third machine learning model.

[0080] In some embodiments, the feature representation receiving module 610 is further configured to: receive encoded information from a client, the encoded information being obtained by performing data encoding operations on at least the feature representation and prompt information, the data encoding operations including encryption operations and / or compression operations; and perform data decoding operations on the encoded information to obtain at least the feature representation and prompt information, the data decoding operations including decryption operations and / or decompression operations.

[0081] The units and / or modules included in device 500 and / or device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 500 and / or device 600 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0082] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices can, for example, be used to implement... Figure 1 The client 110 and / or server 120 in the middle.

[0083] Figure 7 A block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to achieve Figure 1 The client 110 and / or the server 120.

[0084] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processors 710 or processing units, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processor 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0085] Electronic device 700 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 700.

[0086] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0087] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0088] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0089] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0090] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0091] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0092] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0094] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for model inference, the method being implemented on a client side, the method comprising: In response to receiving model input, a feature representation of the model input is determined using a first machine learning model deployed locally on the client, wherein the first machine learning model is part of a second machine learning model; At least the feature representation and prompt information are sent to the server to continue processing the feature representation using the second machine learning model, wherein the prompt information at least indicates the proportion of the first machine learning model in the second machine learning model; and The model output corresponding to the model input is obtained from the server, and the model output is generated by the second machine learning model by processing the feature representation.

2. The method according to claim 1, further comprising: Determine the local computing power of the client; The proportion of the first machine learning model in the second machine learning model to be deployed to the client is determined based on the local computing power. as well as The first machine learning model is deployed on the client based on the stated percentage.

3. The method according to claim 1, wherein sending the feature representation and prompt information to the server at least includes: At least the feature representation and the prompt information are subjected to data encoding operations to obtain encoded information, the data encoding operations including encryption operations and / or compression operations; as well as The encoded information is sent to the server.

4. The method of claim 1, wherein the second machine learning model includes a model structure of an encoder and a decoder, the encoder includes multiple encoder layers, and wherein the first machine learning model includes at least one encoder layer among the multiple encoder layers.

5. The method of claim 1, wherein the second machine learning model comprises a decoder-only model structure, the decoder comprising a plurality of decoder layers, and wherein the first machine learning model comprises at least one of the plurality of decoder layers.

6. A method for model inference, the method being implemented on a server, the method comprising: The client receives at least a feature representation and a prompt, wherein the feature representation is obtained by processing the model input by a first machine learning model deployed locally on the client, and the prompt indicates at least the proportion of the first machine learning model in the second machine learning model; Using the second machine learning model, the feature representation is processed based on the prompt information to determine the model output corresponding to the model input; as well as The model output is sent to the client.

7. The method according to claim 6, wherein the server is deployed with the second machine learning model or the third machine learning model, and the third machine learning model is the model portion of the second machine learning model other than the first machine learning model.

8. The method of claim 7, wherein processing the feature representation based on the prompt information using the second machine learning model comprises: The prompt word input for the second machine learning model or the third machine learning model is determined based at least on the feature representation and the prompt information; as well as The prompt word input is provided to the second machine learning model or the third machine learning model to process the feature representation using the second machine learning model or the third machine learning model.

9. The method of claim 6, wherein receiving at least feature representation and prompt information from the client comprises: The client receives encoded information, which is obtained by performing data encoding operations on at least the feature representation and the prompt information, including encryption and / or compression operations. as well as Perform a data decoding operation on the encoded information to obtain at least the feature representation and the prompt information. The data decoding operation includes a decryption operation and / or a decompression operation.

10. An apparatus for model inference, the apparatus being implemented on a client, the apparatus comprising: A feature representation determination module is configured to, in response to receiving model input, determine the feature representation of the model input using a first machine learning model deployed locally on the client, wherein the first machine learning model is part of a second machine learning model; The feature representation sending module is configured to send at least the feature representation and prompt information to the server to continue processing the feature representation using the second machine learning model, wherein the prompt information at least indicates the proportion of the first machine learning model in the second machine learning model; as well as The model output acquisition module is configured to acquire the model output corresponding to the model input from the server, wherein the model output is generated by the second machine learning model by processing the feature representation.

11. An apparatus for model inference, the apparatus being implemented on a server, the apparatus comprising: The feature representation receiving module is configured to receive at least a feature representation and prompt information from the client, wherein the feature representation is obtained by processing the model input by a first machine learning model deployed locally on the client, and the prompt information indicates at least the proportion of the first machine learning model in a second machine learning model; The feature representation processing module is configured to use the second machine learning model to process the feature representation based on the prompt information in order to determine the model output corresponding to the model input; as well as The model output sending module is configured to send the model output to the client.

12. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the electronic device to perform the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 9.

13. A computer-readable storage medium having stored thereon computer-executable instructions, said computer-executable instructions being executable by a processor to implement the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 9.

14. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 9.

Images