Inference method, apparatus, and computing device cluster

Abstract

A reasoning method applied to a server, comprising: receiving a first request from a client; selecting a first sub-pool from a pool based on a similarity between the first request and at least one sub-pool in the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and at least one inference result of a historical request is contained in a sub-pool; and decoding the first request based on the first sub-pool to obtain a first inference result. In this way, by dividing the pool into multiple sub-pools and considering the association relationship between the subsequent request and the sub-pool, the subsequent request can use more accurate data to improve the reasoning quality. Meanwhile, by using the historical request to create the pool, the subsequent request does not need to re-construct the pool in the cold start stage, which improves the quality of the candidate word generated in the initial stage of the subsequent request, and the pool can be independent of the offline database.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a reasoning method, apparatus and computing device cluster. Background Technology

[0002] Large language models (LLMs) represent a significant advancement in natural language processing. Trained using deep learning techniques, they are capable of understanding and generating near-human-level text. However, as the model size increases, the performance bottleneck of autoregressive decoding becomes increasingly apparent, particularly in terms of memory bandwidth. Insufficient memory access speed leads to increased latency and limits the parallel processing capabilities of AI accelerators. To address this issue, parallel decoding techniques have been proposed to improve the efficiency of autoregressive decoding. Parallel decoding utilizes redundant computing power for parallel prediction and verification by simultaneously predicting multiple subsequent tokens, effectively reducing the number of inference steps required in the entire decoding process and significantly improving the inference efficiency of LLMs. However, the output quality of large models still needs improvement during the decoding process. Summary of the Invention

[0003] This application provides a reasoning method, apparatus, computing device cluster, computer storage medium, and computer product that can improve the output quality of models.

[0004] Firstly, this application provides a reasoning method applied to a server, comprising: receiving a first request from a client; selecting a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and each sub-pool contains at least one historical request and its reasoning result; and decoding the first request based on the first sub-pool to obtain a first reasoning result. For example, the pool can be computer-recognizable and stored in the computer in the form of some data structure, such as an array, list, hash table, etc. For example, the pool can be an N-gram pool.

[0005] In this way, during the inference process, by dividing the pool into multiple sub-pools and considering the relationships between subsequent requests and sub-pools, subsequent requests can use more accurate data, improving inference quality. Simultaneously, by creating pools using historical requests, subsequent requests do not need to rebuild the pool during the cold start phase, improving the quality of candidate tokens generated in the initial stage of subsequent requests. Furthermore, since the pools are created based on historical requests, they do not rely on offline databases.

[0006] In one possible implementation, after obtaining the first inference result, the method further includes adding the first inference result to the first sub-pool. This populates the sub-pool with data, increasing the number of selectable data items and thus providing more candidate tokens for subsequent requests.

[0007] In one possible implementation, the first sub-pool is selected from the pool based on the similarity between the first request and at least one sub-pool contained in the pool. This includes calculating the distance between the first request and the cluster centers associated with the pool, where a cluster center is the cluster center of historical requests associated with a sub-pool, and the sub-pool associated with the cluster center with the shortest distance is the first sub-pool. Thus, the first sub-pool can be selected from the pool through similarity calculation.

[0008] In one possible implementation, the first request is decoded based on a first sub-pool, including: in any decoding step, multiple candidate sequences are obtained based on the input sequence of that decoding step and the tokens contained in the first sub-pool, wherein a candidate sequence is obtained by combining the input sequence and at least one token from the first sub-pool; in any decoding step, a large model is used to process the multiple candidate sequences to obtain at least one token following the input sequence, wherein the large model performs reasoning only once for tokens with repeated prefixes in different candidate sequences. Thus, during the large model decoding process, when multiple candidate sequences have duplicate content, the duplicate content can be avoided, saving computational resources.

[0009] In one possible implementation, the method further includes updating the sub-pool partitioning within the pool based on the first request. This ensures that subsequent requests always match the latest clustering result.

[0010] In one possible implementation, the method further includes: when receiving requests from different clients, and the sub-pools selected from the pool for the requests from different clients are the same, grouping the requests from different clients into a batch; and decoding the requests from different clients based on the same sub-pool corresponding to the requests from different clients. This allows similar requests to share a single sub-pool, thereby increasing the number of selectable data in the sub-pool and further improving the quality of candidate tokens.

[0011] In one possible implementation, before selecting the first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, the method further includes: if the number of historical requests reaches a preset value, clustering the historical requests to divide the pool into multiple sub-pools, where the number of sub-pools is the preset value. This achieves the division of sub-pools. Specifically, when the number of historical requests does not reach the preset value, conventional methods can be used to infer the historical requests to obtain the corresponding inference results. When the number of historical requests reaches the preset value, clustering the historical requests then yields the corresponding number of sub-pools.

[0012] Secondly, this application provides an inference apparatus deployed on a server, comprising: a receiving module and a processing module. The receiving module is configured to receive a first request from a client; the processing module is configured to filter out a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and each sub-pool contains at least one historical request and its inference result; the processing module is further configured to decode the first request based on the first sub-pool to obtain a first inference result.

[0013] In one possible implementation, after obtaining the first inference result, the processing module is further configured to add the first inference result to the first sub-pool.

[0014] In one possible implementation, when the processing module selects the first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, it specifically calculates the distance between the first request and the cluster center associated with the pool, wherein a cluster center is the cluster center of the historical requests associated with a sub-pool, and the sub-pool associated with the cluster center with the shortest distance is the first sub-pool.

[0015] In one possible implementation, when the processing module decodes the first request based on the first sub-pool, it specifically performs the following: In any decoding step, based on the input sequence of any decoding step and the tokens contained in the first sub-pool, it obtains multiple candidate sequences, wherein a candidate sequence is obtained by combining the input sequence and at least one token in the first sub-pool; In any decoding step, it uses a large model to process the multiple candidate sequences to obtain at least one token after the input sequence, wherein the large model performs reasoning only once for tokens with repeated prefixes in different candidate sequences.

[0016] In one possible implementation, the processing module is also used to: update the division of sub-pools in the pool based on the first request.

[0017] In one possible implementation, the processing module is further configured to: when receiving requests from different clients, and the sub-pools selected from the pool for the requests from different clients are the same, group the requests from different clients into a batch, and decode the requests from different clients based on the same sub-pool corresponding to the requests from different clients.

[0018] In one possible implementation, before the processing module selects the first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, it is further configured to: cluster the historical requests to divide the pool into multiple sub-pools if the number of historical requests reaches a preset value, wherein the number of sub-pools is a preset value.

[0019] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0021] Fifthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of an inference system provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the workflow of an inference system provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram illustrating a clustering of requests provided in an embodiment of this application;

[0026] Figure 4This is a flowchart illustrating a reasoning method provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram illustrating the execution process of a certain decoding step in a large model provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a reasoning device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0032] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0033] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0034] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0036] First, the relevant technical terms involved in the technical solution provided in this application will be introduced.

[0037] (1) Token

[0038] In natural language processing, a token is a basic unit in text. A token can be a word, subword, character, or any other element that can express the semantics of the text.

[0039] (2) Autoregressive Decoding

[0040] Autoregressive decoding is a text generation technique that progressively constructs sentences or paragraphs by predicting the next token in a given context. In this process, each newly generated token depends on previously generated tokens. Because autoregressive decoding generates tokens one by one, it ensures that the generation of each token fully considers the preceding contextual information, thus improving the accuracy and coherence of the generated text.

[0041] (3) Server

[0042] In large model service deployments, the server refers to the physical or virtual machine that hosts and runs large models. It is responsible for handling requests from clients, performing model inference, and returning results. Servers typically need high-performance computing capabilities, especially when dealing with large machine learning models, which may require graphics processing units or other dedicated hardware acceleration.

[0043] (4) Client

[0044] In a large-scale service-oriented deployment, the client is the component that directly interacts with the user. It can be a desktop application, mobile application, web application, or web-based application, etc. The client is responsible for collecting user input (such as text messages or voice commands) and sending it to the server over the network, as well as receiving responses from the server and displaying the results returned by the server.

[0045] (5) Agent

[0046] An agent is an entity capable of autonomous decision-making and action to achieve a specific goal. In large-scale model service deployments, agents typically utilize advanced artificial intelligence technologies such as large language models, possessing powerful natural language processing and understanding capabilities. They are able to perceive and understand their environment and take actions based on this information to accomplish predetermined objectives. Agents can run independently of servers or be integrated into servers. When an agent runs independently of a server, it can be deployed on computing resources other than the server. In this embodiment, both the agent and the server can be deployed on the cloud side.

[0047] Next, the technical solution provided in this application will be introduced.

[0048] Generally, parallel decoding techniques typically involve two phases: a pool construction phase and a pool verification phase. In the pool construction phase, multiple candidate tokens can be generated in parallel and initially filtered. In the pool verification phase, the filtered candidate tokens are thoroughly verified, and the optimal token is selected as the final output. This multi-stage parallel processing significantly improves the inference efficiency and output quality of LLM. However, in a service-oriented inference environment, the pool is empty whenever a new request is received, requiring the generation of candidate tokens from scratch. This results in a low proportion of candidate tokens that meet expectations in the initial stage, i.e., a low hit rate, thus affecting the model's output quality. For example, the pool can be computer-recognizable and stored in the computer in the form of some data structure, such as an array, list, or hash table. For instance, the pool could be an N-gram pool.

[0049] Additionally, to improve the quality of candidate tokens generated in the initial stage, the pool is constructed using an offline database. Since the offline database stores a large number of high-quality tokens, the quality of candidate tokens generated in the initial stage can be improved. However, this method is highly dependent on the offline database, and the quality of the data in the offline database significantly impacts the model's output quality.

[0050] In view of this, this application provides an inference scheme that creates a pool using historical requests, eliminating the need to rebuild the pool during the cold start phase for subsequent requests and improving the quality of candidate tokens generated at the initial stage of subsequent requests. Furthermore, since the pool is created based on historical requests, it does not rely on an offline database. Simultaneously, the pool is divided into multiple sub-pools, and the relationships between subsequent requests and sub-pools are considered, allowing subsequent requests to use tokens contained in more accurate sub-pools, thus improving inference quality. The inference scheme is described in detail below.

[0051] For example, Figure 1 A schematic diagram of the architecture of an inference system provided in an embodiment of this application is shown. Figure 1 As shown, the inference system includes a terminal side and a cloud side. At least one client 110 is deployed on the terminal side. A word segmenter 111 can be configured in client 110. The word segmenter 111 is responsible for converting string data into integer data so that the model can process it. It should be understood that the word segmenter 111 can also be configured on the cloud side, depending on the actual situation, and is not limited here.

[0052] Server 120 and Agent 130 are deployed on the cloud side. Server 120 can be configured with a pool maintenance module 121 and a similarity matching module 122.

[0053] The pool maintenance module 121 is primarily responsible for building and maintaining the pool, as well as dividing it into sub-pools. For example, a pool may contain tokens that can serve as other requests. During the initial pool construction, the pool maintenance module 121 can use algorithms such as cosine similarity, term frequency-inverse text frequency, latent semantic analysis, and K-Means to cluster historical requests from the client side, for example, clustering the questions contained in historical requests to obtain multiple clusters. After clustering, the inference results of all requests contained in a cluster can be considered as a sub-pool, thus constructing the pool. The number of clusters can be preset. A cluster can be associated with a sub-pool within the pool. A sub-pool can include at least one historical request and its inference result. The data in the sub-pool can be referred to as "drafts," and the tokens contained in these drafts can serve as tokens for other requests. In some embodiments, when the number of requests from the client side is greater than or equal to the preset number of clusters, the pool maintenance module 121 can be triggered to cluster the requests to construct the pool.

[0054] After the pool is constructed, the pool maintenance module 121 can store the inference results of subsequent requests into the sub-pools associated with those requests. Specifically, for each sub-pool associated with a subsequent request, the distance between the subsequent request and the cluster centers of each group associated with that sub-pool can be calculated by the similarity matching module 122. Furthermore, as the number of requests from the client increases, the initial clustering method may no longer meet the grouping requirements of subsequent pools. Therefore, the pool maintenance module 121 can update the clustering results, for example, by splitting, merging, updating, or discarding already divided sub-pools, so that new requests always match the latest clustering results.

[0055] The similarity matching module 122 is primarily responsible for calculating the similarity between new requests from the client and sub-pools, filtering out the sub-pools with the highest similarity to the new request, and using these sub-pools to decode the new request. Specifically, when server 120 receives a new request from the client, the similarity matching module 122 can use methods such as Euclidean distance, cosine distance, or edit distance to calculate the distance between the new request and the cluster centers of the clusters associated with each sub-pool in the draft pool. After completing the distance calculation, the similarity matching module 122 can establish an association between the new request and the sub-pool associated with the cluster center closest to the new request, so that this sub-pool can be used for inference about the new request. For example, when a pool contains sub-pools 1 and 2, where sub-pool 1 is associated with cluster 1 and sub-pool 2 is associated with cluster 2, after server 120 receives a new request Q from the endpoint, it can calculate the distance d1 between the new request Q and the center of cluster 1, and the distance d2 between the new request Q and the center of cluster 2. When d1 is less than d2, an association can be established between the new request Q and sub-pool 1. Additionally, the similarity matching module 122 can group requests with close relevance into a batch, and send the requests in each batch and their corresponding sub-pools to Agent 130. This allows candidate tokens to be selected from sub-pools associated with the request during inference.

[0056] Agent 130 can be configured with a decoding module 131 and an inference engine 132.

[0057] The decoding module 131 is primarily responsible for selecting drafts from the sub-pool associated with the request during the current decoding step of a given request and passing them to the inference engine 132. Specifically, for any decoding step of a request, the decoding module 131 can first use similarity algorithms such as cosine distance to select a certain number of drafts for the current input sequence from the sub-pool associated with that request. For example, suppose the current input sequence is "The cat sat on the," and we need to predict the next token. In this case, we can select three drafts corresponding to the current input "the" from the corresponding sub-pool. For example, if the selected drafts are: Draft 1: "gray mat on the floor," Draft 2: "tree in the garden," and Draft 3: "use the gray mat."

[0058] The inference engine 132 is primarily responsible for performing inference tasks to obtain the inference results for each request. For any decoding step of a request, the inference engine 132 can merge the drafts selected by the decoding module 131 for the current input sequence into the current verification branch to further verify these drafts, calculate the probability of each token in the draft in the current context, and select the optimal candidate token. Next, the inference engine 132 can merge the selected optimal candidate token into the current forward branch to form a candidate sequence, and perform a final forward propagation on the candidate sequence to ensure that the selected optimal candidate token is the most suitable token in the current context. Finally, the inference engine 132 can predict the token in the next decoding step. When predicting the token in the next decoding step, the finally selected candidate sequence is the input sequence for the next decoding step. At this time, the decoding module 131 can continue to use it to select the required drafts from the corresponding sub-pool. For example, in the current decoding step, assuming the current input sequence is "The cat sat on the", at least one token needs to be predicted. At this point, three drafts corresponding to the current input "The cat sat on the" can be selected from the corresponding sub-pools. Let's assume the following are selected: Draft 1: "gray mat on the floor", Draft 2: "tree in the garden", and Draft 3: "use the gray mat that is in the room". Validate each token in these three drafts. The optimal candidate tokens for Draft 1 are determined to be "gray" and "mat", for Draft 2 to be "tree", and for Draft 3 to be "gray", "mat", and "that". Merging "gray" and "mat" from Draft 1 into the current forward branch yields candidate sequence 1: "The cat sat on the gray mat". Merging "gray" and "mat" from Draft 2 into the current forward branch yields candidate sequence 2: "The cat sat on the tree". Merging "gray", "mat", and "that" from Draft 3 into the current forward branch yields candidate sequence 3: "The cat sat on the gray mat that". Suppose that after forward propagation through these three candidate sequences, "gray" and "mat" are determined to be the most suitable tokens. At this point, it can be predicted that the next decoding step will proceed, namely, predicting at least one token following the new input sequence "The cat sat on the gray mat".It should be understood that when the selected optimal candidate token is not the most suitable token, the inference engine 132 can perform inference on the input sequence that does not contain the optimal candidate token in order to find a more suitable token.

[0059] In the aforementioned inference system, by creating a pool using historical requests, subsequent requests do not need to rebuild the pool during the cold start phase, thus improving the quality of candidate tokens generated in the initial stage of subsequent requests. Furthermore, since the pool is created based on historical requests, it does not rely on an offline database. Simultaneously, by dividing the pool into multiple sub-pools and considering the relationships between subsequent requests and sub-pools, subsequent requests can use more accurate data, improving inference quality. Moreover, when there are multiple subsequent requests, similar requests can share a sub-pool, which increases the number of drafts available in the sub-pool, further improving the quality of candidate tokens.

[0060] In the aforementioned inference system, for each request from the client, the number of drafts selected from the sub-pool during decoding can be enormous in each decoding step. Each draft's optimal candidate token and the input sequence can form a candidate sequence, further increasing the number of candidate sequences. A candidate sequence can be obtained by combining the input sequence with at least one token from a draft. When there are multiple candidate sequences, they may frequently contain repeated prefixes, such as "in range" and "in sorted," both containing the prefix "in." Directly feeding these to a large model for validation would result in redundant computation of the same prefixes, wasting computational resources. Therefore, to avoid wasting computational resources, when there are multiple candidate sequences composed of the optimal candidate token and the input sequence, the large model can perform inference only once for tokens with repeated prefixes in different candidate sequences, thereby reducing redundant computation and saving computational resources. For example, when the candidate sequences are determined to be "The cat is very friendly and" and "The cat is very playful and", only one inference needs to be performed for "The cat is very". The inference result can be "friendly" or "playful", or other tokens.

[0061] When identifying repeating prefixes in candidate sequences, a trie (prefix tree) can be built based on these candidate sequences. Each node in the trie, up to the root node, forms a prefix in the candidate sequence, and the weight of a node represents the frequency of that prefix in the candidate sequence. It is observed that prefixes with higher frequency are actually obtained from the subtree formed by the nodes with the highest weights. By performing a breadth-first traversal on this subtree, a pseudo-sentence can be obtained. Due to the properties of the trie, the candidate sequences are always subsequences of this pseudo-sentence. By designing a mask matrix, the original dependencies of the tokens in the pseudo-sentence can be restored during large-scale model validation. Finally, if multiple tokens in the candidate sequence are accepted by the large model, the inference speed of the large model is improved. In some embodiments, the acceptance strategy used here may be, but is not limited to, accepting only when the tokens in the candidate sequence are the same as the tokens that the large model should have generated.

[0062] The workflow of the above reasoning system can be described as follows: Figure 2 As shown. In Figure 2 In this process, the client sends a request to the server. After receiving the request from the client, the server can put the request into a request queue.

[0063] Once the number of requests in the request queue reaches a certain level, the server can perform clustering operations on the requests. Specifically, for example... Figure 3 As shown, after a new request arrives, the server calculates the distance between the new request and the cluster center. This distance can be calculated using Euclidean distance, cosine distance, or edit distance methods. Each new request is associated with the nearest sub-pool and can be added to the corresponding cluster. This determines the sub-pool that each new request should use. Figure 3 In this context, a draft token refers to the set of inference results corresponding to a request within each cluster. As the number of requests increases, the centers of the already divided clusters will gradually change. Therefore, it is necessary to update the previous clustering results. At this time, the already divided sub-pools can be split, merged, updated, or discarded. In this way, the questions in new requests can always match the latest clustering results. For example, sub-pool updates can be performed, but are not limited to, upon receiving a new request.

[0064] exist Figure 2In this process, after clustering requests, the server can associate each request with a sub-pool, and a sub-pool can be associated with at least one request. Next, the server can batch-schedule requests based on the correlation between different requests. Specifically, the server can group requests associated with the same sub-pool into a batch and send each batch of requests and its corresponding sub-pool to the Agent for decoding. In some embodiments, request decoding can include three stages: preprocessing, inference, and post-processing. Preprocessing can include format conversion operations such as normalization, scaling, data augmentation, and encoding conversion, and can also include selecting the optimal candidate tokens from the sub-pools and forming a candidate sequence. Inference can include processing the candidate sequence to obtain at least one token following a given input sequence. Post-processing can include sampling, penalty terms, temperature control, and whether to use a greedy strategy to select the most suitable output token from the probability matrix output by the model, improving the diversity and quality of the generated text. Sampling is the process of selecting the next word from the vocabulary distribution predicted by the model; the penalty term is used to control the complexity of the model and reduce overfitting; the temperature parameter is used to control the randomness in the sampling process; and the greedy strategy selects the most likely result at each step to generate the final result.

[0065] exist Figure 2 In this process, after receiving a sub-pool and corresponding requests from the server, for any batch of requests and its corresponding sub-pool, the Agent can, in each decoding step, select the optimal candidate token from that sub-pool for the input sequence related to these requests. Furthermore, it combines the selected optimal candidate token with the corresponding input sequence to form a candidate sequence, and determines the output of the current decoding step based on the formed candidate sequence to proceed to the next decoding step. The process of selecting the optimal candidate token from the sub-pool and forming a candidate sequence can be called "preprocessing." Figure 2 In a large model, the inference process can include a prefill stage and a decoding stage. The input to the prefill stage includes a candidate sequence consisting of candidate tokens and the input sequence. The initial input to the decode stage can include the output of the prefill stage and a candidate sequence consisting of candidate tokens associated with that output. The inference processes for these two stages are described above. Figure 1 The descriptions of the decoding module 131 and the inference engine 132 in the previous section will not be repeated here. For example, if the Agent receives requests from multiple batches simultaneously, it will process them according to the system's default batch processing method (such as first-in-first-out inference or inference of similar length questions first).

[0066] exist Figure 2In the process, after the large model completes inference, the Agent can return the inference results of each request to the server and client. Upon receiving the inference results, the server can store them in the sub-pool containing the request for that inference result; the client, upon receiving the inference results, can display them to the user. For example, the client can also use a tokenizer configured on it to transform the inference results from the Agent before displaying the transformed inference results to the user.

[0067] The above is an introduction to the inference system and its workflow provided in the embodiments of this application. For ease of understanding, the following uses inference on the GSM8K dataset with llama2-7B as a practical example to explain the above model inference scheme in detail. In this embodiment, under the large model service-oriented inference scenario, parallel decoding is used to fully utilize the computing power of AI accelerators (such as graphics processing units, GPUs), thereby improving the overall throughput. The number of sub-pools is K.

[0068] 1) Construct the pool

[0069] At this point, the historical requests are empty. When the length of the request queue is equal to N (N≥K), the K-Means algorithm is used to cluster each request into K sub-pools. The cluster center matrix is ​​recorded as P, and the number of data points corresponding to each cluster center (i.e., the cluster size) is also recorded. Here, the number of rows in the cluster center matrix P is K, and the number of columns in the cluster center matrix P is the length of the features in the request. The number of inference results corresponding to each request in each sub-pool is recorded as Q. The number of inference results in each sub-pool can be the same or different.

[0070] After constructing the sub-pools, reasoning is performed on each request to obtain the reasoning results for each request. Furthermore, the relationship between the requests contained in the sub-pools and their corresponding reasoning results is established, thereby constructing the draft within the sub-pools.

[0071] 2) Update the pool

[0072] When M new requests are received, the sub-pools within the main pool are updated. During the update, the cluster center matrix P and the features of the M requests are concatenated (vertically) to form a new feature matrix P'. Q and a matrix of length M consisting entirely of ones are concatenated to form a new weight matrix Q'. The K-Means algorithm with weights Q' is then used to cluster the feature matrix P' to obtain K sub-pools. This method of updating the pools avoids the exponential growth in clustering algorithm time as the number of requests increases.

[0073] 3) Batch scheduling

[0074] Questions belonging to the same sub-pool from M requests are grouped into batches and then passed to the Agent for model inference. For any given sub-pool, a certain number of inference results can be selected and sent to the Agent for inference. For example, when a sub-pool is associated with 3 requests from the M requests, these 3 requests can be grouped into a batch, and in each decoding step, 9 inference results can be selected from this sub-pool and sent as candidate sequences to the Agent for inference.

[0075] 4) Parallel Decoding

[0076] After receiving a batch of requests and the corresponding candidate sequences, the agent can use parallel decoding algorithms such as lookahead to decode them. During decoding, a radix-tree can be used to construct a tree-like input from subsequences with the same shared prefix.

[0077] Next, based on the above content, a reasoning method provided by an embodiment of this application will be introduced.

[0078] For example, Figure 4 A flowchart illustrating an inference method provided in an embodiment of this application is shown. It is understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by an inference device, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in an electronic device or server; typically, it can be configured on a cloud computing platform. Figure 4 As shown, the reasoning method may include the following steps:

[0079] S401, Receive the first request from the client.

[0080] In this embodiment, after receiving information such as the question input by the user, the client can transmit a request containing the relevant information such as the question input by the user to the server so that the server can process the request.

[0081] S402. Based on the similarity between the first request and at least one sub-pool contained in the pool, a first sub-pool is selected from the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and a sub-pool contains at least one historical request and the reasoning result of that historical request.

[0082] In this embodiment, a pool can be configured on the server side. This pool can consist of multiple sub-pools. For example, the pool configured on the server side can be computer-recognizable and stored in the computer in the form of some data structure, such as an array, list, hash table, etc. For example, the pool can be an N-gram pool. The sub-pools within the pool are obtained by clustering historical requests received by the server. For example, clustering historical requests using the K-means algorithm. A sub-pool can contain at least one historical request and its inference result. Since the sub-pools are obtained by clustering historical requests, the correlation between the data in a sub-pool is relatively high. For example, an inference result in a sub-pool can be understood as a draft. In some embodiments, when the number of historical requests has not reached the preset number of sub-pools, the server can continue to count the number of historical requests. When the number of historical requests reaches the requirement, the historical requests are then clustered to obtain sub-pools.

[0083] Furthermore, after receiving the first request, the server can use algorithms such as cosine similarity or Euclidean distance to calculate the similarity between the first request and each sub-pool contained in the pool. In this embodiment, after clustering historical requests, each sub-pool can be associated with the cluster center (i.e., the cluster center) of a cluster. When calculating the similarity between the first request and the sub-pool, the distance between the first request and each cluster center can be calculated as the similarity between the first request and each sub-pool. Then, the sub-pool associated with the cluster center with the shortest distance can be taken as the first sub-pool. In some embodiments, considering that the cluster centers corresponding to historical requests may change as requests increase, in this embodiment, the server can also update the division of sub-pools in the pool based on the first request. For example, the first request can be treated as a historical request and re-clustered, or a weighted K-means algorithm can be used to re-cluster, so that subsequent requests can always match the latest clustering results.

[0084] S403. Based on the first sub-pool, decode the first request to obtain the first inference result.

[0085] In this embodiment, after selecting the first sub-pool required for the first request, the first request can be decoded based on the first sub-pool to obtain the first inference result. For example, both the first sub-pool and the first request can be input into the large model for decoding. When decoding, the large model can use parallel decoding techniques such as lookahead or autoregressive decoding techniques, depending on the actual situation; no limitation is made here. Furthermore, after obtaining the first inference result, it can be added to the first sub-pool to populate the data in the first sub-pool.

[0086] As one possible implementation, in any decoding step during decoding, such as Figure 5 As shown, in S501, based on the input sequence of any decoding step and the tokens contained in the first sub-pool, multiple candidate sequences are obtained. A candidate sequence is obtained by combining the input sequence and at least one token from the first sub-pool. For example, the similarity between the input sequence of any decoding step and each inference result in the first sub-pool can be calculated using algorithms such as cosine similarity. Multiple inference results are then selected based on the calculated similarity, for example, selecting the top k inference results. Then, the optimal candidate token is selected from the inference results and combined with the input sequence to form a candidate sequence. For example, at least one optimal candidate token can be selected from one inference result. In S502, in any decoding step, a large model is used to process multiple candidate sequences to obtain at least one token following the input sequence. The large model only performs inference once for tokens with repeated prefixes in different candidate sequences. After obtaining the candidate sequences, the large model can process these candidate sequences to determine whether the tokens in the inference results of historical requests contained in each candidate sequence are accepted. In this way, during the decoding process of a large model, when multiple candidate sequences have repeated prefixes, the repeated prefixes can be processed again without repeating them, thus saving computational resources.

[0087] In this way, during the inference process, by dividing the pool into multiple sub-pools and considering the relationships between subsequent requests and sub-pools, subsequent requests can use more accurate data, improving inference quality. Simultaneously, by creating pools using historical requests, subsequent requests do not need to rebuild the pool during the cold start phase, improving the quality of candidate tokens generated in the initial stage of subsequent requests. Furthermore, since the pools are created based on historical requests, they do not rely on offline databases.

[0088] In some embodiments, when requests are received from different clients, and the sub-pools selected from the pool for these requests from different clients are the same, the server can group these requests from different clients into a batch and decode these requests based on the same sub-pool they correspond to. This allows similar requests to share a sub-pool, thereby increasing the amount of selectable data in the sub-pool and further improving the quality of candidate tokens.

[0089] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments described above can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0090] Next, based on the methods in the above embodiments, an inference device provided in this application will be described.

[0091] For example, Figure 6 A schematic diagram of an inference device according to an embodiment of this application is shown. Exemplarily, this inference device can be, but is not limited to, deployed on a server. For example... Figure 6 As shown, the inference device 600 includes a receiving module 601 and a processing module 602. The receiving module 601 receives a first request from a client. The processing module 602 filters out a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool. The sub-pools in the pool are obtained by clustering historical requests, and each sub-pool contains at least one historical request and its inference result. The processing module 602 further decodes the first request based on the first sub-pool to obtain a first inference result.

[0092] In some embodiments, after obtaining the first inference result, the processing module 602 is further configured to: add the first inference result to the first sub-pool.

[0093] In some embodiments, when the processing module 602 selects a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, it is specifically used to: calculate the distance between the first request and the cluster center associated with the pool, wherein a cluster center is the cluster center of the historical requests associated with a sub-pool, and the sub-pool associated with the cluster center with the shortest distance is the first sub-pool.

[0094] In some embodiments, when the processing module 602 decodes the first request based on the first sub-pool, it is specifically used to: in any decoding step, obtain multiple candidate sequences based on the input sequence of any decoding step and the tokens in the first sub-pool, wherein a candidate sequence is obtained by combining the input sequence and at least one token in the first sub-pool; in any decoding step, process the multiple candidate sequences using a large model to obtain at least one token after the input sequence, wherein the large model performs reasoning only once for tokens with repeated prefixes in different candidate sequences.

[0095] In some embodiments, the processing module 602 is further configured to: update the division of sub-pools in the pool based on the first request.

[0096] In some embodiments, the processing module 602 is further configured to: when receiving requests from different clients, and the sub-pools selected from the pool for the requests from different clients are the same, group the requests from different clients into a batch, and decode the requests from different clients based on the same sub-pool corresponding to the requests from different clients.

[0097] In some embodiments, before the processing module 602 selects the first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, it is further configured to: cluster the historical requests to divide the pool into multiple sub-pools when the number of historical requests reaches a preset value, wherein the number of sub-pools is a preset value.

[0098] In some embodiments, Figure 6 Both the receiving module 601 and the processing module 602 shown can be implemented in software or in hardware. For example, the implementation of the receiving module 601 will be described below. Similarly, the implementation of the processing module 602 can be referenced from the implementation of the receiving module 601.

[0099] As an example of a software functional unit, the receiving module 601 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0100] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0101] As an example of a hardware functional unit, the receiving module 601 may include at least one computing device, such as a server. Alternatively, the receiving module 601 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0102] The multiple computing devices included in the receiving module 601 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 601 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 601 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0103] It should be noted that, in other embodiments, the receiving module 601 can be used to execute any step in the reasoning method described in the above embodiments, and the processing module 602 can also be used to execute any step in the reasoning method described in the above embodiments. Furthermore, the steps implemented by the receiving module 601 and the processing module 602 can also be specified as needed, and different steps in the reasoning method described in the above embodiments can be implemented by the receiving module 601 and the processing module 602 respectively. Figure 6 The inference device 600 shown has all the functions.

[0104] This application also provides a computing device 700. For example... Figure 7 As shown, the computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0105] The 702 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 704 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 704 may include a path for transmitting information between various components of the computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0106] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0107] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0108] The memory 706 stores executable program code, and the processor 704 executes the executable program code to implement the aforementioned functions respectively. Figure 6 The receiving module 601 and processing module 602 shown herein perform their functions to implement the reasoning method described in the above embodiments. That is, the memory 706 stores instructions for executing the reasoning method described in the above embodiments.

[0109] Alternatively, the memory 706 stores executable code, and the processor 704 executes the executable code to implement the aforementioned functions respectively. Figure 6 The inference device 600 shown functions to implement the inference method described in the above embodiments. That is, the memory 706 stores instructions for executing the inference method described in the above embodiments.

[0110] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0111] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0112] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same instructions for executing the inference method described in the above embodiments.

[0113] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the inference method described in the above embodiments. In other words, a combination of one or more computing devices 700 can jointly execute instructions for executing the inference method described in the above embodiments.

[0114] It should be noted that the memory 706 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 6 The inference device 600 shown contains some of its functions. That is, the instructions stored in the memory 706 in different computing devices 700 can implement the functions of one or more modules in the receiving module 601 and the processing module 602.

[0115] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, two computing devices 700A and 700B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 706 in computing device 700A stores instructions for performing the functions of the receiving module 601. Simultaneously, the memory 706 in computing device 700B stores instructions for performing the functions of the processing module 602.

[0116] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0117] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different in that the memory 706 of one or more computing devices 700 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0118] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the aforementioned inference method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions for executing the aforementioned inference method.

[0119] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0120] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0121] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0122] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0123] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0124] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0125] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A reasoning method, characterized in that, Applied to the server side, the method includes: Receive the first request from the client; Based on the similarity between the first request and at least one sub-pool contained in the pool, a first sub-pool is selected from the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and each sub-pool contains at least one historical request and the reasoning result of the historical request; Based on the first sub-pool, the first request is decoded to obtain the first inference result.

2. The method according to claim 1, characterized in that, After obtaining the first reasoning result, the process also includes: Add the first inference result to the first sub-pool.

3. The method according to claim 1 or 2, characterized in that, The step of selecting a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool includes: Calculate the distance between the first request and the cluster center associated with the pool, where a cluster center is the cluster center of a historical request associated with the sub-pool. The sub-pool associated with the cluster center that has the shortest distance is the first sub-pool.

4. The method according to any one of claims 1-3, characterized in that, The step of decoding the first request based on the first sub-pool includes: In any decoding step, based on the input sequence of the decoding step and the tokens contained in the first sub-pool, multiple candidate sequences are obtained, wherein a candidate sequence is obtained by combining the input sequence and at least one token in the first sub-pool; In any of the decoding steps, the large model is used to process the multiple candidate sequences to obtain at least one token following the input sequence, wherein the large model performs reasoning only once for tokens with repeated prefixes in different candidate sequences.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on the first request, update the division of sub-pools within the pool.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: When requests are received from different clients, and the sub-pools selected from the pool for the requests from different clients are the same, the requests from different clients are grouped into a batch, and the requests from different clients are decoded based on the same sub-pool corresponding to the requests from different clients.

7. The method according to any one of claims 1-6, characterized in that, Before selecting a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, the method further includes: When the number of historical requests reaches a preset value, the historical requests are clustered to divide the pool into multiple sub-pools, wherein the number of sub-pools is the preset value.

8. A reasoning device, characterized in that, Deployed on the server side, including: The receiving module is used to receive the first request from the client; The processing module is configured to filter out a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, wherein the sub-pools in the pool are obtained by clustering historical requests, and each sub-pool contains at least one historical request and the reasoning result of the historical request. The processing module is further configured to decode the first request based on the first sub-pool to obtain a first inference result.

9. The apparatus according to claim 8, characterized in that, After obtaining the first reasoning result, the processing module is further configured to: Add the first inference result to the first sub-pool.

10. The apparatus according to claim 8 or 9, characterized in that, When the processing module selects a first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, it is specifically used for: Calculate the distance between the first request and the cluster center associated with the pool, wherein a cluster center is the cluster center of historical requests associated with a sub-pool, and the sub-pool associated with the cluster center with the shortest distance is the first sub-pool.

11. The apparatus according to any one of claims 8-10, characterized in that, When the processing module decodes the first request based on the first sub-pool, it is specifically used for: In any decoding step, based on the input sequence of the decoding step and the tokens contained in the first sub-pool, multiple candidate sequences are obtained, wherein a candidate sequence is obtained by combining the input sequence and at least one token in the first sub-pool; In any of the decoding steps, the large model is used to process the multiple candidate sequences to obtain at least one token following the input sequence, wherein the large model performs reasoning only once for tokens with repeated prefixes in different candidate sequences.

12. The apparatus according to any one of claims 8-11, characterized in that, The processing module is further configured to: Based on the first request, update the division of sub-pools within the pool.

13. The apparatus according to any one of claims 8-12, characterized in that, The processing module is further configured to: When requests are received from different clients, and the sub-pools selected from the pool for the requests from different clients are the same, the requests from different clients are grouped into a batch, and the requests from different clients are decoded based on the same sub-pool corresponding to the requests from different clients.

14. The apparatus according to any one of claims 8-13, characterized in that, Before filtering out the first sub-pool from the pool based on the similarity between the first request and at least one sub-pool contained in the pool, the processing module is further configured to: When the number of historical requests reaches a preset value, the historical requests are clustered to divide the pool into multiple sub-pools, wherein the number of sub-pools is the preset value.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-7, wherein the cluster of computing devices includes at least one computing device.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-7, wherein the computing device cluster includes at least one computing device.

Images