Data processing method based on large model and electronic device

Abstract

The present disclosure relates to a data processing method and an electronic device. The method comprises: receiving a cache request from a client, the cache request comprising prefix information; providing a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database; receiving a processing request from the client, the processing request comprising prompt information and the prompt information having the prefix information; obtaining the embedding vector of the prefix information pre-stored in the vector database; and obtaining a model output by using a large model based on the embedding vector of the prefix information and other information in the prompt information except the prefix information. By this scheme, when the large model processes the prompt information, repeated processing of the prefix information can be avoided, thereby reducing the cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computers, and more particularly to a data processing method and an electronic device. BACKGROUND

[0002] A generative large language model is a deep learning model capable of generating natural language text. Such models are based on complex neural network architectures, such as transformers, and are pre-trained on large amounts of text data to learn the statistical properties and patterns of language.

[0003] A user can invoke a large language model through a prompt, such as a prompt word. The user can input a request including the prompt word through an Application Interface (API) of a client to guide the large language model to reply. The prompt word in different requests can have a repeated sequence, and the processing of the prompt word by the large language model each time can result in a high computational cost. SUMMARY

[0004] According to an example embodiment of the present disclosure, a data processing scheme is provided, in which an embedding vector of prefix information can be pre-stored, so that the large language model can avoid repeated processing of the prefix information when processing the prompt information, thereby reducing the cost.

[0005] In a first aspect of the present disclosure, a data processing method is provided. The method comprises: receiving a caching request from a client, the caching request including prefix information; providing a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database; receiving a processing request from the client, the processing request including prompt information and the prompt information having the prefix information; obtaining the embedding vector of the prefix information pre-stored in the vector database; and obtaining a model output by using a large model based on the embedding vector of the prefix information and other information in the prompt information except the prefix information.

[0006] In a second aspect of the present disclosure, an electronic device is provided. The electronic device comprises: at least one processing unit; at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which when executed by the at least one processing unit, cause the electronic device to perform the method described in the first aspect of the present disclosure.

[0007] In a third aspect of the disclosure, a data processing apparatus is provided. The apparatus comprises: a receiving unit configured to receive a cache request from a client, the cache request comprising prefix information; a providing unit configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database; the receiving unit is further configured to receive a processing request from the client, the processing request comprising hint information, and the hint information having the prefix information; a vector obtaining unit configured to obtain the embedding vector of the prefix information pre-stored in the vector database; and an output unit configured to obtain a model output based on the embedding vector of the prefix information and other information in the hint information except the prefix information, using a large model.

[0008] In a fourth aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon machine executable instructions which, when executed by a device, cause the device to perform the method described according to the first aspect of the disclosure.

[0009] In a fifth aspect of the disclosure, a computer program product is provided, comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method described according to the first aspect of the disclosure.

[0010] In a sixth aspect of the disclosure, an electronic device is provided, comprising: processing circuitry configured to perform the method described according to the first aspect of the disclosure.

[0011] The summary is provided to introduce a selection of concepts that are further described below in the of the Invention. This summary is not intended to identify key or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. Other features will be more fully described in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other features, aspects, and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numbers represent like elements throughout. In the drawings:

[0013] Figure 1 A schematic block diagram of a system according to some embodiments of the disclosure is shown;

[0014] Figure 2 A flowchart of an example process according to some embodiments of the disclosure is shown;

[0015] Figure 3 A schematic diagram of a communication process according to some embodiments of the disclosure is shown;

[0016] Figure 4 a block diagram illustrating an example apparatus in accordance with some embodiments of the present disclosure; and

[0017] Figure 5 a block diagram illustrating an example apparatus that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted in a limited sense as set forth in the embodiments set forth herein, but rather, the embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] In embodiments of the present disclosure, a large model can be referred to as a large language model (LLM), a generative large language model, a transformer-based large language model, etc., which are not limited by the present disclosure. The large model in embodiments of the present disclosure can be applied to various application scenarios, such as writing articles, generating dialogues, creating poems, generating content for social media / news, generating code / annotations / documents, generating teaching materials, assisting language learning, etc.

[0020] A large language model can be trained on a large amount of text data to be able to understand and generate natural language text. The large language model can be based on a transformer architecture, which can effectively handle long-range dependencies through a self-attention mechanism and has excellent performance in parallel computing. The large language model can include multiple transformer layers, for example, a transformer layer can be an encoder layer or a decoder layer. The transformer layer can include a self-attention layer, a feedforward neural network (FNN) layer, and a layer normalization (LN) layer.

[0021] A user can input a prompt through an API of a client device (such as a mobile phone, a tablet computer, a desktop computer, a wearable device, etc.) to call a large model. An API is a set of predefined rules and protocols for building and interacting with interfaces between software applications. The API defines the data format of the request and the structure of the response, so that different applications can understand and exchange data with each other. Through the API, an application can provide its functions or services for other applications to use without sharing its source code. Developers can use APIs to integrate third-party services into their own applications or extend the functionality of the application.

[0022] Exemplarily, components of an API can include endpoints, requests, and responses. An endpoint can be a Uniform Resource Locator (URL) address in an API that can be accessed. A request can include data sent by a user to an API, including a method (such as GET, POST, PUT, DELETE), a path, parameters, and a body. A response can be data returned by an API to a client, including a status code, header information, and a body.

[0023] Specifically, a user can input prompt information through an API, which can include, for example, a string of characters that the user wants to ask. The API can also perform the number of maximum reply tokens and formats, etc. Further, a large model can generate a reply string according to the prompt information and provide it to a client device.

[0024] The running cost of a large model is related to the number of tokens. A large model based on a transformer architecture can utilize a full attention mechanism to consider the relevance of all other tokens when processing each token. For example, each token needs to interact with other tokens to calculate attention weights. The computational complexity of this self-attention mechanism is the square of the number of tokens. When the input sequence is long, the computational cost of full attention is too high. The more tokens there are, the more computation is required, and the higher the running cost is. In addition, each token has its own vector representation in the model, which needs to be stored. In fact, the number of tokens is one of the important indicators to measure the processing workload of the model. Therefore, in the case of processing more tokens, more storage resources are needed, more computing resources are consumed, and more time is spent.

[0025] There can be repeated sequences of characters in the user request input to the large model, which can be referred to as “common prefix” or “prefix information” in the embodiments of the present disclosure. For example, in a question and answer system, the user may input a question that includes a word group such as “please ask”, “please tell me”, “I want to know” every time. For another example, the context of some questions can be the same long text sequence, such as a question and answer system for a novel A, and all the text content of the novel A can be considered as a common prefix. In this case, if the large model processes the prompt information completely every time, it will cause repeated processing of the common prefix, resulting in high computational cost and low processing efficiency.

[0026] To solve the above problems and potential other problems, embodiments of the present disclosure provide a data processing scheme for a large model. In this scheme, the embedding vector of the prefix information can be stored, so that when processing the prompt information, the stored embedding vector of the prefix information can be obtained, avoiding repeated processing of the prefix information, which can improve the processing efficiency.

[0027] Figure 1 A schematic block diagram of a system 100 is shown in accordance with some embodiments of the present disclosure. As shown, the system 100 includes a client 110 and a server 120, where the server 120 can include a large model 122 and a vector database 124. Figure 1

[0028] In embodiments of the present disclosure, a user can use the client 110 to access via an API interface between the client 110 and the server 120. Illustratively, the user can obtain access to the vector database 124 by subscription or the like, for example, the user can access the size, period, access rate, etc. of the vector database 124 depending on the user’s permission. By way of example, the user can apply for access to the vector database 124 based on their own cost control strategy or the like.

[0029] Illustratively, the client 110 can be implemented as an electronic device on the user side, such as a mobile phone, a tablet computer, a desktop computer, a wearable device, a vehicle-mounted device, etc.

[0030] Illustratively, the server 120 can be implemented as a remote device or can be located in the cloud, for example, the server 120 can be implemented as an electronic device or a distributed cluster, etc. It should be understood that although Figure 1 Although the server 120 is shown to include the large model 122 and the vector database 124, in some embodiments, the server 120 can be implemented in a distributed manner, for example, the server 120, the large model 122, and the vector database 124 can be physically located in different locations.

[0031] The embedding vectors of the prefix information can be stored in the vector database 124. It can be understood that multiple embedding vectors of multiple pieces of prefix information can be stored in the vector database 124, for example, the embedding vectors of “please ask”, “please tell me”, “I want to know”, etc. in the question and answer system are stored in the vector database 124. Illustratively, the multiple embedding vectors of the multiple pieces of prefix information can be stored based on a caching request from the user (from the client 110).

[0032] The prefix information and the embedding vectors stored in the vector database 124 can also have corresponding caching priority information. As an example, the stored information can be as shown in Table 1. It should be noted that different prefix information can have the same or different caching priorities, depending on the user’s settings.

[0033] Table 1

[0034] Cache priority 1 Prefix information 1 Embedding vector 1 Cache priority 2 Prefix information 2 Embedding vector 2 Cache priority 3 Prefix information 3 Embedding vector 3 … … …

[0035] ​Exemplarily, different indexes can also be assigned to different prefix information, and stored together with the embedding vectors. As an example, the stored information can be as shown in Table 2. It should be noted that the indexes of different prefix information are different, and the assignment of indexes is not necessarily continuous.

[0036] Table 2

[0037] Index 1 Prefix information 1 Embedding vector 1 Cache priority 1 Index 2 Prefix information 2 Embedding vector 2 Cache priority 2 Index 3 Prefix information 3 Embedding vector 3 Cache priority 3 … … … …

[0038] It can be understood that the content and format of the stored information shown in Table 1 or Table 2 above are only illustrative, and do not constitute a limitation on the embodiments of the present disclosure. Alternatively, the index information and the cache priority information can be combined into one field for storage. For example, “001010” is stored corresponding to the prefix information 1, the first three bits “001” indicate that the index is 1, and the last three bits “010” indicate that the cache priority is level 2. The present disclosure does not limit this.

[0039] It can be understood that the vector database can also be referred to as other names, such as storage module, memory, cache area, or similar terms, etc., and the present disclosure does not limit this.

[0040] The user can input prompt information through the client 110. The prompt information input by the user can include prefix information and other information. Then the large model 122 can obtain the embedding vector of the prefix information from the vector database 124, and then process it together with other information to obtain the output of the model. Exemplarily, the model output can be provided or displayed to the user via the client 110. In this way, at least the process of determining the embedding vector of the prefix information can be saved, and the computing cost is saved.

[0041] Figure 2 A flowchart of a processing process 200 according to an example embodiment of the present disclosure is shown. In block 210, a cache request from a client is received, and the cache request includes prefix information. In block 220, a first response is provided to the client, and the first response indicates that the embedding vector of the prefix information has been stored in the vector database. In block 230, a processing request from the client is received, and the processing request includes prompt information and the prompt information has the prefix information. In block 240, the embedding vector of the prefix information pre-stored in the vector database is obtained. In block 250, based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, a large model is used to obtain a model output.

[0042] Exemplarily, the prompt information includes a plurality of tokens. A token can be a minimum processing unit of input and output text in natural language processing and large language models, for example, a token can be a character, a word, a subword, a special symbol, etc. Exemplarily, when processing text, a large model can perform a tokenization process on the text, thereby splitting the string in the text into a series of tokens. For example, the prompt information is input to the large model, and the large model can split the prompt information into a plurality of tokens through the tokenization process.

[0043] The prefix information can include one or more tokens. The prefix information can be part of the prompt information, and different prompt information can have the same or different prefix information. Exemplarily, different prompt information can include the same prefix information, i.e., a common prefix. The common prefix can be used to set the context of the dialogue, and the large model can understand the intent of the following dialogue or text based on the common prefix. Based on the common prefix, the large model can be guided to provide a specific type of output (i.e., an answer), such as interrogative, imperative, or explanatory. In embodiments of the present disclosure, as described in conjunction with Figure 1

[0044] In some implementations, the large model is a large language model based on a transformer architecture, and the stored embedding vectors of the prefix information can be accelerated for calculation using a key-value pair cache (KV cache). The key-value pair can include the key (Key) and value (Value) of the attention mechanism, and the cache can refer to storing both mathematical vectors of the key and value into memory or hard disk.

[0045] In embodiments of the present disclosure, the server 120 can provide an API of the large model so that the user obtains access to the large model, such as obtaining the right to use and control the vector database 124, etc. Exemplarily, the user can send an API request through the client 110. The server 120 can determine the vector database accessible by the client 110 based on the API request. Optionally, the API request can include an API key, an access method, a request body, etc. For example, the access method can be a Hypertext Transfer Protocol (HTTP) method, etc. Exemplarily, after that, the user can use the client to use the large model through the API. Optionally, the API can also be referred to as an API endpoint, for example, it can be a URL address.

[0046] ​The process of the embodiments of the present disclosure can include two stages: a first stage and a second stage. In some implementations, the first stage can be used to confirm whether the embedding vector of the prefix information has been cached, and the second stage can be used to request the large model to provide the model output corresponding to the prompt information.

[0047] In the first stage, a caching request from the client can be received, which can include a cache control parameter. Illustratively, the cache control parameter can include prefix information. Optionally, the cache control parameter can further include cache priority information. For example, the caching request can include the prefix information to be stored (e.g., text XXXXX) and the cache priority corresponding to the prefix information (e.g., Lx). Optionally, the caching request can include a first execution code to indicate the type of the caching request. Optionally, the first execution code can represent that the request is a request of the first stage. For example, the first execution code can be 0, indicating that the request is a caching request of the first stage.

[0048] In some examples, based on the caching request, the embedding vector of the prefix information can be determined using the large model, and the embedding vector of the prefix information can be stored in the vector database. Optionally, the embedding vector of the prefix information can be determined using the key-value pair cache of the transformer-based large model, for example, the forward propagation algorithm of the transformer neural network can be called to determine the embedding vector of the prefix information.

[0049] In the embodiments of the present disclosure, the cache priority information can indicate the duration of storage of the prefix information. Illustratively, assuming that N levels of cache priority are preset, N different storage duration thresholds can be correspondingly set. The correspondence or mapping relationship between the cache priority and the storage duration threshold can be predefined. For prefix information of a certain cache priority, if the storage time length of the prefix information has reached the storage duration threshold corresponding to the cache priority, the embedding vector of the prefix information can be cleared from the cache. In this way, long-term invalid occupation of the cache can be avoided, and sufficient storage space can be ensured for the embedding vectors of other prefix information, thereby improving the utilization rate of the cache.

[0050] In the embodiments of the present disclosure, the cache priority information can indicate the order of removal from the cache. Illustratively, if the occupancy rate of the cache has exceeded a threshold (e.g., 90% or other values) or the idle rate is lower than a threshold (e.g., 10% or other values), the embedding vector of the prefix information with a lower cache priority can be cleared from the cache. In this way, the prefix information with a high cache priority can be ensured to be retained in the cache for as long as possible, and repeated processing of the prefix information with a high cache priority can be avoided.

[0051] In embodiments of the present disclosure, the embedding vector of one or more prefix information can be purged in combination with one or more of the following factors: available storage space of the vector database, length of time stored, cache priority information, size of occupied storage space, frequency of being called or accessed within a historical predetermined period, etc. For example, among the prefix information with the lowest cache priority, the prefix information with the earliest previous access time can be removed first.

[0052] In some other examples, after receiving the cache request, it can be determined whether the embedding vector of the prefix information has been stored, i.e., whether the embedding vector of the prefix information exists in the vector database. If the embedding vector of the prefix information does not exist in the vector database, the embedding vector can be determined and stored through the above process. If the embedding vector of the prefix information already exists in the vector database, the above process does not need to be repeated, which can save processing amount and improve efficiency. For example, the embedding vector of the prefix information has been stored in the vector database based on historical cache requests, but the user does not know whether it has been purged, so the user can initiate a cache request again to ensure that the embedding vector of the prefix information is included in the vector database.

[0053] In some other examples, if the embedding vector of the prefix information does not exist in the vector database, and the vector database does not have enough space to store the embedding vector of the prefix information, it can be further determined whether to store the embedding vector of the prefix information. For example, if the cache priority of another prefix information already existing in the vector database is lower than the cache priority of the prefix information, the embedding vector of the prefix information with the lower cache priority can be removed and the embedding vector of the prefix information can be stored. For example, if the cache priority of all prefix information already existing in the vector database is higher than the cache priority of the prefix information, the embedding vector of the prefix information is not stored.

[0054] In some examples, a first response to the cache request can be sent to the client. The first response can indicate whether the embedding vector of the prefix information exists in the vector database. Optionally, the first response can include a cache status code to indicate whether the embedding vector of the prefix information has been stored in the vector database. For example, the cache status code can be a Boolean value, such as 0 or 1, where 1 indicates that it has been stored and 0 indicates that it has not been stored. Optionally, the first response can also include a status code to indicate whether the cache request is successful.

[0055] In the second stage, a processing request can be received from the client, and the processing request can include the prompt information. Optionally, the processing request can include a second execution code to indicate the type of the processing request. Optionally, the second execution code can indicate that the request is a request of the second stage. For example, the second execution code can be 1, indicating that the request is a processing request of the second stage. It can be understood that the second execution code is different from the first execution code.

[0056] For example, after receiving the foregoing first response, the client 110 can determine whether to send a processing request. For example, it can be determined whether to perform the second stage based on the cost control strategy of the user.

[0057] For example, after receiving the processing request, the prefix information in the prompt information can be extracted or determined. Then, the embedding vector of the prefix information can be obtained or extracted from the vector database. In this way, for the prefix information in the prompt information, the stored embedding vector can be directly obtained from the vector database without further processing by the large model, saving computing resources and improving processing effect.

[0058] Further, the model output corresponding to the prompt information can be obtained by the large model, and the model output can be sent to the client 110. For example, the second response to the processing request can be sent to the client 110, and the second response can include the model output. For example, the model output includes a reply or an answer to the input text.

[0059] In addition, optionally, the cache request or the processing request in the embodiments of the present disclosure can have a corresponding request priority. For example, requests with higher request priorities can be processed faster to avoid excessive delay. It can be understood that the server 120 in the embodiments of the present disclosure can perform reasonable and effective resource management. For example, in the process of receiving and processing requests, it is determined that the resources of the server are effectively managed to meet the quality of service requirements corresponding to the request, wherein the resources of the server can include central processing unit (CPU), graphics processing unit (GPU), memory, etc.

[0060] Figure 3 A schematic diagram of a communication process 300 according to some embodiments of the present disclosure is shown. The process 300 can involve the client 110 and the server 120, wherein the client 110 can be operated or used by a user.

[0061] At 302, the client 110 can send an API request to request to establish the API endpoint 115. For example, a user can input the API request through a browser or an application. The API request can include an API key, an HTTP method, etc. For example, the API request can be used to integrate the API component into an application of the client 110, so as to facilitate the user to use the large model through the application.

[0062] The API component can include API endpoints, requests, and responses. The API endpoints can be one or more, which can be used to receive requests from the client 110 and return responses.

[0063] The client 110 can send a request to the API endpoint, such as the aforementioned cache request, processing request, etc. The request can include an API key for authentication. The request can include an HTTP method, which indicates that the request is a POST request, i.e., the data in the request (cache request, processing request) needs to be transmitted to the server. The request can include a request body, which indicates data in a predetermined format. The request body can include indication information of the large model, cache control parameters, prompt information, first / second execution code, etc. For example, the indication information of the large model can include model parameters, such as the name of the model, the version number, etc. In this way, in the case that the server 120 is associated with multiple large models, a specific large model 122 can be quickly located based on the model parameters.

[0064] The server 120 can return a response to the client 110, such as the aforementioned first response, second response, etc. The response can include a status code, which indicates whether the request is successful. The response can include a cache status code, which indicates whether the embedding vector of the prefix information is stored in the vector database. The response can include a response body, which carries the data to be returned. For example, the response body of the second response includes the model output.

[0065] At 304, the client 110 can perform the first-stage communication. The client 110 can send a first-stage request (such as the aforementioned cache request), the request body of which can include cache control parameters and execution code. The cache control parameters can include a common prefix, which can be in the form of a string. The cache control parameters can also include cache priority information, which can be a positive integer from 1 to 5. The execution code can be the first execution code, such as 0, indicating that the request is the first-stage communication. Optionally, the common prefix in the cache request can be user-defined.

[0066] At 306, the API endpoint 115 can send the first-stage request to the server 120. The server 120 can invoke the large model 122 at 308, accelerate the computation with the key-value cache, and thus obtain the embedding vector of the common prefix at 310. Subsequently, the embedding vector of the common prefix is stored in the vector database 124 at 312.

[0067] At 314, the server 120 can return the first-stage response, e.g., the first response as described above. The response can include a status code indicating that the request is successful. The response further includes a cache status code, e.g., a Boolean value 1, indicating that the embedding vector of the common prefix has been stored in the vector database 124. Optionally, the first response can further indicate that the communication between the client 110 and the server 120 has been established or connected.

[0068] Exemplarily, the steps 304 to 314 can be repeated to store more embedding vectors of common prefixes in the vector database 124.

[0069] At 320, the client 110 can perform the second-stage communication. The client 110 can send a second-stage request, e.g., the processing request as described above, and the request body of the request can include the prompt information and the execution code. The execution code can be a second execution code, e.g., 1, indicating that the request is for the second-stage communication. For example, the client 110 can switch the execution code from 0 to 1 to initiate the second-stage communication.

[0070] At 322, the API endpoint 115 can send the second-stage request to the server 120. The server 120 can subsequently obtain the embedding vector of the common prefix in the vector database 124, and further obtain the model output corresponding to the prompt information with the large model 122.

[0071] At 324, the server 120 can return the second-stage response, e.g., the second response as described above. The response can include a status code indicating that the request is successful. The response further includes a response body, e.g., including the model output.

[0072] Additionally, additionally, the cache in the vector database 124 can be managed at 330. Specifically, the embedding vector of the prefix information with a storage time length reaching a corresponding threshold can be removed based on the storage time length threshold associated with the cache priority. Alternatively, the embedding vector of the prefix information with a low priority can be removed to store the embedding vector of the prefix information with a high priority when the storage demand of the prefix information with the high priority is high. It should be understood that the specific management manner of the vector database is not limited by the embodiments of the present disclosure, e.g., a hash table, an ordered set, or the like can be used to implement the management.

[0073] In this way, the embedding vector of the prefix information of the prompt information in the embodiments of the present disclosure is pre-stored in the vector database, and when processing the prompt information, the prefix information no longer needs to be repeatedly processed. In this way, the number of word units processed can be reduced, the length of the processed text is controlled, the use cost of the large model is effectively controlled, and the large model is more efficiently utilized.

[0074] It should be noted that the above describes the main embodiments of the present disclosure, but the present disclosure is not limited to specific implementation details. Illustratively, in the communication process, the requested data packet can be sent to the Internet through the network of the Internet service provider via the Internet service provider. Illustratively, in order to speed up the speed of content transmission, the request can be distributed through a content distribution network. Illustratively, the server 120 shown in the figure can be implemented as a model server, Figure 3 the API endpoint 115 in the figure can be implemented as an API server. Illustratively, the request can be evenly distributed to different servers through a load balancer. For example, if there are multiple model servers, the request can be distributed according to the current load of the multiple model servers, the health status, and the configured policy, etc. For example, the number of servers can be automatically increased or reduced according to the load. For example, the instances of the available multiple model servers can be determined through service discovery. For example, the request can be routed to the instances of the available multiple model servers as needed.

[0075] It should be understood that in the embodiments of the present disclosure, "first", "second", "third" and the like are only to indicate that the plurality of objects can be different, but at the same time do not exclude that the two objects are the same, and should not be interpreted as any limitation on the embodiments of the present disclosure.

[0076] It should also be understood that the manners, cases, categories and divisions of embodiments in the embodiments of the present disclosure are only for the convenience of description and should not constitute a special limitation. The features in various manners, categories, cases and embodiments can be combined with each other as long as they are consistent with logic.

[0077] It should also be understood that the above description is only to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Those skilled in the art can make various modifications or changes or combinations, etc. according to the above description. The schemes after such modifications, changes or combinations are also within the scope of the embodiments of the present disclosure.

[0078] It should also be understood that the above description focuses on the differences between the various embodiments, and the same or similar parts can be referred to or learned from each other. For the sake of brevity, they will not be repeated here.

[0079] Figure 4A schematic block diagram of an example apparatus 400 is shown according to some embodiments of the present disclosure. The apparatus 400 can be implemented by software, hardware, or a combination of both. As shown in FIG. 4, the apparatus 400 includes a receiving unit 410, a providing unit 420, a vector obtaining unit 430, and an output unit 440. Figure 4 The receiving unit 410 is configured to receive a cache request from a client, the cache request including prefix information. The providing unit 420 is configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database. The receiving unit 410 is further configured to receive a processing request from the client, the processing request including prompt information, and the prompt information having the prefix information. The vector obtaining unit 430 is configured to obtain the embedding vector of the prefix information pre-stored in the vector database. The output unit 440 is configured to obtain a model output based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, using a large model.

[0080] The receiving unit 410 is configured to receive a cache request from a client, the cache request including prefix information. The providing unit 420 is configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database. The receiving unit 410 is further configured to receive a processing request from the client, the processing request including prompt information, and the prompt information having the prefix information. The vector obtaining unit 430 is configured to obtain the embedding vector of the prefix information pre-stored in the vector database. The output unit 440 is configured to obtain a model output based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, using a large model.

[0081] The receiving unit 410 is configured to receive a cache request from a client, the cache request including prefix information. The providing unit 420 is configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database. The receiving unit 410 is further configured to receive a processing request from the client, the processing request including prompt information, and the prompt information having the prefix information. The vector obtaining unit 430 is configured to obtain the embedding vector of the prefix information pre-stored in the vector database. The output unit 440 is configured to obtain a model output based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, using a large model.

[0082] The receiving unit 410 is configured to receive a cache request from a client, the cache request including prefix information. The providing unit 420 is configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database. The receiving unit 410 is further configured to receive a processing request from the client, the processing request including prompt information, and the prompt information having the prefix information. The vector obtaining unit 430 is configured to obtain the embedding vector of the prefix information pre-stored in the vector database. The output unit 440 is configured to obtain a model output based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, using a large model.

[0083] Optionally, the cache request further includes cache priority information corresponding to the prefix information.

[0084] The receiving unit 410 is configured to receive a cache request from a client, the cache request including prefix information. The providing unit 420 is configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database. The receiving unit 410 is further configured to receive a processing request from the client, the processing request including prompt information, and the prompt information having the prefix information. The vector obtaining unit 430 is configured to obtain the embedding vector of the prefix information pre-stored in the vector database. The output unit 440 is configured to obtain a model output based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, using a large model.

[0085] Optionally, the size of the vector database is associated with user permissions of the client. Optionally, the cache request includes a first execution code for indicating a type of the cache request. Optionally, the processing request includes a second execution code for indicating a type of the processing request.

[0086] Exemplarily, the apparatus 400 can comprise a clearing unit configured to clear the stored embedding vector of the prefix information based on at least one of: available storage space of the vector database, cache priority information of the prefix information, stored time length of the embedding vector, or frequency of invocation of the embedding vector within a historical predetermined time period.

[0087] Figure 4 The apparatus 400 of the above-described embodiments can be used to implement the above-described process of the above-described embodiments. Figures 1 to 3 For brevity, the above-described process will not be repeated here.

[0088] The division of modules or units in the embodiments of the present disclosure is illustrative only, and merely a logical function division, and actual implementation can have another division manner. In addition, each functional unit in the disclosed embodiments can be integrated in one unit, or can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0089] Figure 5 A block diagram of an example device 500 that can be used to implement embodiments of the present disclosure is shown. It should be understood that Figure 5 The device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. For example, device 500 can be used to perform the above-described Figures 1 to 3 processes described above.

[0090] As Figure 5 shown, the device 500 is in the form of a general-purpose computing device. Components of the computing device 500 can include, but are not limited to, one or more processors or processing units 510, memory 520, storage 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processing unit 510 can be a real or virtual processor and is capable of executing various processing stored in the memory 520 according to programs. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 500.

[0091] The computing device 500 typically includes a plurality of computer storage media. Such media can be volatile, nonvolatile, removable, and / or non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Storage 520 can be volatile memory (such as registers, cache, RAM), non-volatile memory (such as ROM, EEPROM, flash memory), or some combination of volatile and non-volatile memory. Storage 530 can be removable or non-removable media, and can include machine readable media such as flash drives, disk drives, or any other media capable of storing information and / or data (e.g., training data for training) and accessible by the computing device 500.

[0092] The computing device 500 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown, a removable, non-volatile memory such as a floppy disk drive, a magnetic tape drive, an optical disk drive or a flash drive can be provided. In such instances, each can be connected to the bus by one or more data media interfaces. The memory 520 can include a computer program product 525 having one or more program modules configured to carry out the various methods or actions of the various implementations of the present disclosure. Figure 5 The program module 525 can include, among other things, a program for use in implementing the program functionality of the present disclosure. The program module 525 can also include, among other things, a program for use in implementing the program functionality of the present disclosure. The program module 525 can also include, among other things, a program for use in implementing the program functionality of the present disclosure.

[0093] The communication unit 540 enables communications with other computing devices over a communication media. Additionally, the functionality of the components of the computing device 500 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating over a communication connection. Thus, the computing device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0094] Input device 550 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. Output device 560 can be one or more output devices, such as a display, a speaker, a printer, etc. Computing device 500 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc. through communication unit 540, and / or with one or more devices enabling user interaction with computing device 500, such as a keyboard, a mouse, a joystick, etc. and / or with one or more devices enabling communication with one or more other computing devices, such as a modem, a network card, etc. Such communication can be enabled by an Input / Output (I / O) interface (not shown).

[0095] According to an example implementation of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, there is also provided a computer program product tangibly stored on a non-transitory computer-readable medium and comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, there is provided a computer program product having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0096] According to an example implementation of the present disclosure, there is provided a chip or chip system having computer-executable instructions carried thereon, which, when executed by a device or apparatus or processor, enable the method described above.

[0097] Various aspects of the disclosure can be described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0099] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0100] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0101] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being shown and described in terms of certain implementations and overall functions, the implementations are not intended to exclude other implementations or technologies. Alterations and permutations of certain implementations could produce still other implementations that fall within the scope of the implementations. Accordingly, the phrase "consisting essentially of is defined to not include processes, systems, articles, or materials that do not "consist essentially of the recited elements. The phrase "consisting essentially of is defined to include processes, systems, articles, or materials that "consist essentially of the recited elements, as well as such processes, systems, articles, or materials that also have additional, nonessential elements.

Claims

1. A data processing method, comprising: in a first stage, receiving a cache request from a client, the cache request comprising prefix information, wherein the cache request comprises a first execution code indicating a type of the cache request; providing a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database of a server, the first response further indicating that a communication between the client and the server has been established or has been connected, wherein the client obtains a permission to the vector database by a subscription; in a second stage, receiving a processing request from the client, the processing request comprising prompt information, and the prompt information having the prefix information, wherein the processing request comprises a second execution code indicating a type of the processing request; obtaining the embedding vector of the prefix information pre-stored in the vector database; based on the embedding vector of the prefix information and other information in the prompt information except the prefix information, obtaining a model output by using a large model; and providing a second response to the client, the second response comprising the model output for the processing request.

2. The method of claim 1, further comprising: based on the cache request, determining whether the embedding vector of the prefix information is stored in the vector database; and if it is determined that the embedding vector exists in the vector database, generating the first response.

3. The method of claim 1, further comprising: based on the cache request, determining whether the embedding vector of the prefix information is stored in the vector database; and if it is determined that the embedding vector does not exist in the vector database, determining the embedding vector of the prefix information by using the large model.

4. The method of claim 1, wherein the cache request further comprises cache priority information corresponding to the prefix information.

5. The method of claim 4, further comprising: determining a storage duration threshold corresponding to the cache priority information; and if a storage time length of the embedding vector of the prefix information in the vector database reaches the storage duration threshold, clearing the embedding vector of the prefix information from the vector database.

6. The method of claim 1, wherein a size of the vector database is associated with a user permission of the client.

7. The method of claim 1, further comprising: based on at least one of the following, clearing the embedding vector of the stored prefix information: an available storage space of a vector database, cache priority information of the prefix information, a stored time length of the embedding vector, or a frequency of invocation of the embedding vector within a historical predetermined period of time.

8. An electronic device, comprising: at least one processing unit; ​ ​ ​ ​ at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 7.

9. A data processing apparatus comprising: a receiving unit configured to receive a cache request from a client in a first stage, the cache request comprising prefix information, wherein the cache request comprises a first execution code indicating a type of the cache request; a providing unit configured to provide a first response to the client, the first response indicating that an embedding vector of the prefix information has been stored in a vector database of a server, the first response further indicating that a communication between the client and the server has been established or has been connected, wherein the client obtains a permission to the vector database by a subscription manner; the receiving unit is further configured to receive a processing request from the client in a second stage, the processing request comprising hint information, and the hint information having the prefix information, wherein the processing request comprises a second execution code indicating a type of the processing request; a vector obtaining unit configured to obtain the embedding vector of the prefix information pre-stored in the vector database; an output unit configured to obtain a model output by using a large model based on the embedding vector of the prefix information and other information in the hint information except the prefix information; and the providing unit is further configured to provide a second response to the client, the second response comprising the model output for the processing request.

10. A computer readable storage medium having stored thereon a computer program, the program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images