Method for determining model output result and related device

Abstract

The invention discloses a method for determining a model output result and a related device, and belongs to the field of data processing. The method comprises the steps of obtaining current input information of a generation model; based on the current input information, index structures of multiple computing nodes are retrieved, the index structures are used for indicating historical information of a generation model, the historical information comprises historical input information and / or historical output results, and the multiple computing nodes store intermediate operation results corresponding to the historical information indicated by the respective index structures; and if the retrieval result indicates that part of information in the current input information hits the index structure of at least one computing node, determining a current output result of the generation model based on an intermediate operation result corresponding to the part of information in the at least one computing node. According to the method and the device, the intermediate operation results stored in different computing nodes can be fully utilized, so that reuse of the intermediate operation results in a cross-network scene is realized, the throughput of model reasoning is improved, and the reasoning cost is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method and related apparatus for determining model output results. Background Technology

[0002] Generative models, such as large language models (LLMs) and multimodal models, are used to determine the corresponding output based on user input. For example, in an LLM, the user input is a question, and the LLM can determine the corresponding question-and-answer result based on the question. The process by which a generative model determines its output can be called the inference process of the generative model. To reduce inference costs and improve inference efficiency, some intermediate computation results in the generative model's inference process can be stored, making it easier to reuse these intermediate computation results in subsequent inference processes.

[0003] Currently, generative models are primarily deployed on a single computing node, and the intermediate computation results are reused only within that single node. However, with the widespread deployment of inference services, generative models may be deployed across multiple computing nodes. In this case, how to fully utilize the intermediate computation results stored on different computing nodes has become a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides a method and related apparatus for determining model output results, which can fully utilize intermediate computation results stored in different computing nodes, thereby enabling the reuse of intermediate computation results across network scenarios, improving model inference throughput, and reducing inference costs. The technical solution is as follows:

[0005] In a first aspect, a method for determining the output result of a model is provided, wherein the model is deployed on multiple computing nodes; the method includes: obtaining the current input information of the model; based on the current input information, retrieving the index structure of the multiple computing nodes, wherein the index structure is used to indicate the historical information of the model, the historical information including historical input information and / or historical output results, and the multiple computing nodes storing intermediate computation results corresponding to the historical information indicated by their respective index structures; if the retrieval result indicates that a portion of the current input information matches the index structure of at least one of the multiple computing nodes, then the current output result of the model is determined based on the intermediate computation results corresponding to the portion of information in the at least one computing node.

[0006] In this application, based on the current input information of the generative model, the index structure of multiple computing nodes is retrieved to determine whether the current input information matches the index structure of these multiple computing nodes. If the index structure of some computing nodes is matched, the intermediate computation results corresponding to the matched information can be obtained from these computing nodes without needing to compute the intermediate computation results corresponding to these information. This is equivalent to replacing the computation process with retrieval, storage, and transmission. Since computation costs are often much higher than transmission costs, this method can save computational costs and thus improve inference efficiency. Furthermore, by reusing the intermediate computation results stored by these multiple computing nodes across a network (e.g., a wide area network), the overall scheduling of these multiple computing nodes is achieved, improving overall scheduling efficiency, thereby increasing the throughput of model inference and reducing inference costs.

[0007] The input information can take many forms, such as text, voice, images, and video. The first computing node can be any one of the multiple computing nodes, the one closest to the user terminal, or the one with the lightest load.

[0008] Historical input information refers to the input information used to generate the model before the current model inference. Historical output results refer to the output results determined by the model generated before the current model inference. Historical input information includes one or more input information entries, and historical output results include one or more output results.

[0009] Since these multiple computing nodes are deployed with the same generative model, and each node can receive input information provided by the user for the generative model, and perform inference through the generative model to obtain the output result, each computing node has an index structure that indicates the historical information of the generative model. This historical information includes historical input information and / or historical output results. Therefore, for each of these multiple computing nodes, the index structure indicates the historical information acquired by that computing node, which consists of the historical input information acquired by that computing node and / or the historical output results determined by that computing node through the generative model.

[0010] The index structure of a computing node can be constructed using the historical information acquired by that computing node. For example, for ease of description, any piece of information in the historical information acquired by the computing node is referred to as the target information. The target information can be either input information or output results. Based on the tokens included in the target information, and according to the dependencies of these tokens within the target information, a corresponding index structure is generated.

[0011] The tokens included in the target information are obtained by dividing the target information. For example, if the target information is text, it can be segmented into multiple word units, each word unit being a token. If the target information is an image, it can be divided into multiple image blocks, each image block being a token. If the target information is video, each video frame can be divided into multiple image blocks, each image block being a token.

[0012] For tokens included in the target information, the dependencies between these tokens within the target information are used to indicate the structure of the target information, or in other words, to indicate the position of these tokens within the target information. For example, for text information, if the text information includes multiple tokens, the non-first token depends on each token preceding it. For an image, if the image includes multiple tokens, the non-first token depends on each token preceding it. That is, the non-first token depends on each token preceding it.

[0013] In one possible implementation, the index structure includes multiple tokens, each token corresponding to an intermediate computation result; retrieving the index structure of the multiple computing nodes based on the current input information includes: determining multiple target tokens included in the current input information; and retrieving the index structure of the multiple computing nodes based on the multiple target tokens and according to the dependency relationship of the multiple target tokens in the current input information.

[0014] Based on the above description, for any given piece of information, the dependencies of the tokens included within that information indicate the positions of those tokens within that information. In other words, there are certain dependencies between these tokens. Therefore, based on these multiple target tokens and their dependencies within the current input information, the index structure of these multiple computing nodes can be retrieved using prefix matching, thereby improving retrieval accuracy.

[0015] In one possible implementation, the method is applied to a first computing node among the plurality of computing nodes; before retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: obtaining the index structure of each of the other computing nodes among the plurality of computing nodes besides the first computing node.

[0016] In one possible implementation, the method is applied to a first computing node among the plurality of computing nodes, wherein the at least one computing node does not include the first computing node; determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: obtaining the intermediate computation results corresponding to the partial information from the at least one computing node; and determining the current output result through the generative model based on the intermediate computation results corresponding to the partial information.

[0017] In one possible implementation, the number of the at least one computing node is 1, and it is a second computing node, which fully stores the intermediate calculation results corresponding to the partial information; obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: obtaining the intermediate calculation results corresponding to the partial information from the second computing node.

[0018] Based on the above description, the index structure includes multiple tokens, each token corresponding to an intermediate computation result, and this information may include one or more tokens. Therefore, the second computing node completely storing the intermediate computation results corresponding to this information means that the second computing node stores the intermediate computation results corresponding to each token included in this information.

[0019] In one possible implementation, the number of the at least one computing node is greater than 1, and at least one of the at least one computing node completely stores the intermediate operation results corresponding to the partial information; obtaining the intermediate operation results corresponding to the partial information from the at least one computing node includes: selecting a computing node from the at least one computing node as a second computing node, the second computing node completely storing the intermediate operation results corresponding to the partial information; and obtaining the intermediate operation results corresponding to the partial information from the second computing node.

[0020] Since the number of at least one computing node is greater than 1, meaning that the index structure of at least two computing nodes hits this part of the information, and among these computing nodes, some computing nodes may not have completely stored the intermediate operation results corresponding to this part of the information, while some computing nodes may have completely stored the intermediate operation results corresponding to this part of the information, therefore, the computing node that has completely stored the intermediate operation results corresponding to this part of the information can be selected from the at least one computing node, namely the second computing node, and then the intermediate operation results corresponding to this part of the information can be obtained from the second computing node.

[0021] In one possible implementation, multiple communication links exist between the second computing node and the first computing node. The generative model includes multiple network layers with a sequential inference order. The intermediate computation results corresponding to the partial information include the intermediate computation results required by the multiple network layers. Obtaining the intermediate computation results corresponding to the partial information from the second computing node includes: obtaining the intermediate computation results corresponding to the partial information from the second computing node based on the inference order of the multiple network layers and the communication quality of the multiple communication links, so that the intermediate computation results required by the network layers with earlier inference order are obtained through communication links with high communication quality, and the intermediate computation results required by the network layers with later inference order are obtained through communication links with low communication quality.

[0022] Since these multiple network layers have a sequential inference order, the intermediate computation results corresponding to this information include the intermediate computation results required by these multiple network layers. Therefore, the intermediate computation results required by the network layers with earlier inference order are obtained through a high-quality communication link, while the intermediate computation results required by the network layers with later inference order are obtained through a low-quality communication link. In this way, the intermediate computation results required by the network layers with earlier inference order can be obtained as quickly as possible. This allows for faster processing of subsequent computations by the network layers with earlier inference order while obtaining the intermediate computation results required by the network layers with later inference order, reducing overall inference latency and thus improving the inference efficiency of the generative model.

[0023] In one possible implementation, the number of the at least one computing node is greater than 1, and at least two of the at least one computing nodes completely store the intermediate calculation results corresponding to the partial information; obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: taking the partial information as target information and obtaining the intermediate calculation results corresponding to the target information from the at least two computing nodes.

[0024] In one possible implementation, the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, at least two computing nodes store intermediate operation results corresponding to the first information, and at least one computing node completely stores the intermediate operation results corresponding to the partial information; obtaining the intermediate operation results corresponding to the partial information from the at least one computing node includes: using the first information as target information, obtaining the intermediate operation results corresponding to the target information from the at least two computing nodes, and obtaining the intermediate operation results corresponding to the second information from the one computing node.

[0025] In one possible implementation, the generative model includes multiple network layers with a sequential inference order, and the intermediate computation results corresponding to the target information include the intermediate computation results required by the multiple network layers. Obtaining the intermediate computation results corresponding to the target information from the at least two computing nodes includes: based on the inference order of the multiple network layers and the communication quality between the first computing node and the at least two computing nodes, obtaining the intermediate computation results corresponding to the target information from the at least two computing nodes, so that the intermediate computation results required by the network layer with the earlier inference order are obtained from the computing node with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained from the computing node with low communication quality.

[0026] Since these multiple network layers have a sequential inference order, the intermediate computation results corresponding to this information include the intermediate computation results required by these multiple network layers. Therefore, the intermediate computation results required by the network layers with higher inference order are obtained through computing nodes with higher communication quality, while those required by the network layers with lower communication quality are obtained through computing nodes with lower inference order. This ensures that the intermediate computation results required by the network layers with higher inference order are obtained as quickly as possible. Consequently, while obtaining the intermediate computation results required by the network layers with higher inference order, subsequent computations can be performed as quickly as possible through the network layers with higher inference order, reducing overall inference latency and improving the inference efficiency of the generative model. Furthermore, obtaining intermediate computation results from at least two computing nodes, rather than all from the same node, improves the efficiency of intermediate computation result acquisition.

[0027] In one possible implementation, the method is applied to a first computing node among the plurality of computing nodes, wherein the at least one computing node includes the first computing node, and the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, the first computing node does not completely store the intermediate operation results corresponding to the partial information, and the first computing node stores the intermediate operation results corresponding to the first information; determining the current output result of the generative model based on the intermediate operation results corresponding to the partial information in the at least one computing node includes: obtaining the intermediate operation results corresponding to the second information from the other computing nodes besides the first computing node among the at least one computing nodes; and determining the current output result through the generative model based on the intermediate operation results corresponding to the partial information.

[0028] In one possible implementation, the number of the at least one computing node is 2, and the at least one computing node further includes a second computing node, which fully stores the intermediate calculation results corresponding to the partial information; obtaining the intermediate calculation results corresponding to the second information from the other computing nodes besides the first computing node in the at least one computing node includes: obtaining the intermediate calculation results corresponding to the second information from the second computing node.

[0029] Since the first computing node stores the intermediate calculation results corresponding to the first information but not the intermediate calculation results corresponding to the second information, while the second computing node fully stores the intermediate calculation results corresponding to that part of the information, the intermediate calculation results corresponding to the second information can be obtained from the second computing node.

[0030] In one possible implementation, the number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate operation results corresponding to the partial information; obtaining the intermediate operation results corresponding to the second information from the other computing nodes besides the first computing node from the at least one computing node includes: selecting one computing node from the at least two computing nodes as a second computing node, the second computing node completely storing the intermediate operation results corresponding to the partial information; and obtaining the intermediate operation results corresponding to the second information from the second computing node.

[0031] In one possible implementation, the number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information; obtaining the intermediate calculation results corresponding to the second information from the other computing nodes besides the first computing node in the at least one computing node includes: obtaining the intermediate calculation results corresponding to the second information from the at least two computing nodes.

[0032] In one possible implementation, the method further includes: storing intermediate computation results obtained from other computing nodes besides the first computing node in the at least one computing node, and updating the index structure of the first computing node.

[0033] In one possible implementation, the method is applied to a scheduling node; before retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: obtaining the index structure of the plurality of computing nodes.

[0034] In one possible implementation, the method is applied to a scheduling node; determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: selecting a computing node from the at least one computing node as a second computing node, the second computing node fully storing the intermediate computation results corresponding to the partial information; sending the current input information to the second computing node, so that the second computing node determines the current output result based on the intermediate computation results corresponding to the partial information through the generative model.

[0035] When the search results indicate that part of the current input information matches the index structure of at least one computing node, a second computing node that fully stores the intermediate calculation results corresponding to that part of the information can be directly selected from the at least one computing node. The current output result is determined through the second computing node. There is no need to transmit the intermediate calculation results corresponding to that part of the information. Only the current input information needs to be transmitted to the second computing node. Compared with the intermediate calculation results corresponding to that part of the information, the transmission cost and reasoning cost can be further reduced.

[0036] In one possible implementation, the method is applied to a scheduling node; determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: determining a third computing node from the plurality of computing nodes; sending the current input information to the third computing node; if the at least one computing node does not include the third computing node, then scheduling the intermediate computation results corresponding to the partial information in the at least one computing node to the third computing node, so that the third computing node determines the current output result based on the intermediate computation results corresponding to the partial information through the generative model.

[0037] In one possible implementation, the partial information includes first information and second information, wherein the second information depends on the first information; after sending the current input information to the third computing node, the method further includes: if the at least one computing node includes the third computing node, the third computing node does not completely store the intermediate operation results corresponding to the partial information, and the third computing node stores the intermediate operation results corresponding to the first information, then the intermediate operation results corresponding to the second information stored by the other computing nodes besides the third computing node in the at least one computing node are scheduled to the third computing node, so that the third computing node determines the current output result based on the intermediate operation results corresponding to the partial information through the generation model.

[0038] When at least one computing node includes a third computing node, and the third computing node does not fully store the intermediate operation results corresponding to the information, the intermediate operation results corresponding to the second information stored by other computing nodes are scheduled to the third computing node, so that the third computing node can fully obtain the intermediate operation results corresponding to the information, and then use the intermediate operation results corresponding to the information to determine the current output result.

[0039] In one possible implementation, after retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: if the retrieval result indicates that all information of the current input information matches the index structure of at least one of the plurality of computing nodes, then obtaining the current output result from the at least one computing node.

[0040] In other words, if the historical input information of the generative model is exactly the same as the current input information, then the historical output result corresponding to the historical input information can be directly obtained from at least one computing node, and this historical output result can be used as the current output result. This eliminates the need for repeated reasoning through the generative model, saving reasoning costs and improving the efficiency of determining the model's output result.

[0041] Secondly, a device for determining model output results is provided, the device having the function of implementing the method behavior described in the first aspect above. The device includes at least one module for implementing the method provided in the first aspect above.

[0042] Thirdly, a computing node is provided, comprising a processor and a memory, the memory for storing a computer program for performing the method provided in the first aspect. The processor is configured to execute the computer program stored in the memory to implement the method described in the first aspect.

[0043] Optionally, the computing node may further include a communication bus for establishing a connection between the processor and the memory.

[0044] Fourthly, a computer-readable storage medium is provided, wherein the storage medium stores instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0045] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0046] The technical effects achieved by the second to fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the reasoning process of a large language model provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of an implementation environment for deploying a generative model, provided in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of a distributed storage cluster provided in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of a non-distributed storage cluster provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of another implementation environment provided in the embodiments of this application;

[0053] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0054] Figure 8 This is a flowchart of a method for determining model output results provided in an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of an image including a token provided in an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of a target index structure provided in an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of another target index structure provided in an embodiment of this application;

[0058] Figure 12 This is a schematic diagram of an index structure provided in an embodiment of this application;

[0059] Figure 13 This is a schematic diagram of an index structure for multiple data centers provided in an embodiment of this application;

[0060] Figure 14 This is a flowchart of another method for determining model output results provided in an embodiment of this application;

[0061] Figure 15 This is a schematic diagram of a device for determining model output results provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0063] Before providing a detailed explanation of the method for determining the model output results provided in the embodiments of this application, the terminology, application scenarios, and implementation environment involved in the embodiments of this application will be introduced first.

[0064] First, the terminology used in the embodiments of this application will be explained.

[0065] 1. Data Center (DC)

[0066] A data center comprises a complex set of facilities used for the centralized processing, storage, transmission, exchange, and management of information. This complex set of facilities typically includes servers, storage devices, switches, routers, firewalls, and other equipment. For example, if this complex set of facilities is configured in each of several different regions, then each of these regions can be called a data center. These regions could be different cities, different districts within the same city, or different buildings. Similarly, if this complex set of facilities is configured in each of several different server rooms, then each of these server rooms can be called a data center. Even a single rack within a server room can be considered a data center, and different racks within the same server room are called different data centers.

[0067] 2. Generative Model

[0068] Generative models are used to determine the corresponding output based on input information. Generative models typically include large language models, multimodal models, and so on. Large language models are used to determine the question-and-answer result based on the question posed by the user. Multimodal models are used to determine the output result in another modality based on input information provided by the user in one modality.

[0069] Here, "modality" refers to the form of data, and multimodal data refers to data in multiple forms. For example, multimodal data can include text, images, videos, and audio. Typically, generative models are based on the Transformer architecture, but they can also be based on other architectures, such as those based on the Mamba architecture.

[0070] 3. Intermediate calculation results

[0071] Generative models generate intermediate computation results during inference. These intermediate results are cached, and some may be used multiple times during inference. The intermediate results may differ depending on the model architecture. For example, for a Transformer-based generative model, which typically employs self-attention, the computation usually involves key and value matrices (K and V matrices). Therefore, intermediate results can include both K and V matrices, collectively referred to as KVcache. Of course, intermediate results can also include other information, such as hidden state. Similarly, if a Mamba-based generative model also uses self-attention, its intermediate results can also include KVcache and hidden state.

[0072] Secondly, the application scenarios involved in the embodiments of this application will be introduced.

[0073] The process by which a generative model determines its output is called the generative model's inference process. This inference process includes a prefill phase and a decoding phase. The prefill phase is also called the full inference phase, and the decoding phase is also called the incremental inference phase. The prefill phase is used to perform calculations on the input information to obtain intermediate results. The decoding phase is used to determine the output based on these intermediate results.

[0074] Take a large language model as an example. Please refer to [the documentation / reference]. Figure 1 , Figure 1In this context, T0-T2 are derived from the user's question, and each of T0-T2 can be considered an input token. During the pre-filling stage, the intermediate computation results corresponding to these multiple input tokens T0-T2 are determined, with the intermediate computation result for input token T0 being t0, for input token T1 being t1, and for input token T2 being t2. During the decoding stage, the first decoding block determines the output token T3 based on the intermediate computation results t0-t2 of the multiple input tokens T0-T2, and the intermediate computation result for output token T3 is t3. At this point, the determined intermediate computation results include t0-t3. The second decoding block determines the output token T4 based on the intermediate computation results t0-t2 of the multiple input tokens T0-T2 and the intermediate computation result t3 of the output token T3, and the intermediate computation result for output token T4 is t4. At this point, the determined intermediate computation results include t0-t4. The third decoding block determines the output token T5 based on the intermediate operation results t0-t2 corresponding to the multiple input tokens T0-T2 and the intermediate operation results t3-t4 corresponding to the output tokens T3-T4. The intermediate operation result corresponding to output token T5 is t5, and the determined intermediate operation results at this point include t0-t5. The fourth decoding block determines the output token T6 based on the intermediate operation results t0-t2 corresponding to the multiple input tokens T0-T2 and the intermediate operation results t3-t5 corresponding to the output tokens T3-T5. The intermediate operation result corresponding to output token T6 is t6, and the determined intermediate operation results at this point include t0-t6. The result composed of the multiple output tokens T3-T6 is determined as the question-and-answer result.

[0075] For example, if a user asks the question "What color is an apple?", the question can be divided into multiple input tokens: "apple", "yes", "what", and "color". In the pre-filling stage, the intermediate computation results corresponding to each of these input tokens are determined. In the decoding stage, the first decoding block determines the output token as "apple" based on the intermediate computation results of these input tokens; the second decoding block determines the output token as "of" based on the intermediate computation results of these input tokens and the output token "apple"; the third decoding block determines the output token as "color" based on the intermediate computation results of these input tokens and the output tokens "apple", "of", and "color"; the fourth decoding block determines the output token as "yes" based on the intermediate computation results of these input tokens and the output tokens "apple", "of", "color", and "yes"; and the fifth decoding block determines the output token as "red" based on the intermediate computation results of these input tokens and the output tokens "apple", "of", "color", and "yes". Combining these output tokens yields the question-and-answer result "The color of an apple is red".

[0076] In the above Figure 1 During the inference process, the intermediate computation results corresponding to each input token determined in the pre-filling stage are used by each decoding block in the decoding stage to generate output tokens. Simultaneously, the intermediate computation results corresponding to each output token are also used by subsequent decoding blocks to generate subsequent output tokens. Therefore, a series of intermediate computation results are generated during the inference process, and these intermediate computation results are used multiple times during the inference process.

[0077] To reduce inference costs while maximizing overall throughput, this paper provides an implementation environment for deploying generative models. Please refer to [reference needed]. Figure 2 This implementation environment includes a control node and an inference cluster. The inference cluster comprises a pre-filling cluster and a decoding cluster. The pre-filling cluster includes multiple pre-filling nodes, and the decoding cluster includes multiple decoding nodes. The computation process in the pre-filling phase is scheduled to the pre-filling cluster. After the pre-filling phase is completed, the intermediate computation results are transmitted to the decoding cluster for subsequent computations. This implementation environment can also be called a PD-separated implementation environment. PD separation means that the pre-filling phase and the decoding phase are executed separately in different clusters.

[0078] In typical current PD separation architectures, the transfer of intermediate computation results is a crucial operation. Therefore, a storage cluster can be introduced on top of the traditional PD separation architecture. This storage cluster consists of multiple storage nodes and is used to store the intermediate computation results generated during inference. In this way, the decoding stage can directly obtain the required intermediate computation results from the storage cluster without needing to pre-fill the cluster before transmitting them to the decoding cluster.

[0079] This storage cluster can be either a distributed cluster or a non-distributed cluster. Please refer to [link / reference]. Figure 3 This storage cluster is a distributed cluster, and its multiple storage nodes come from the storage media in the pre-filled cluster and the decoding cluster, such as... Figure 3 The storage media within the dashed elliptical box constitute a distributed storage cluster. Please refer to [reference needed]. Figure 4 This storage cluster is a non-distributed cluster, independent of the pre-filled cluster and the decoding cluster.

[0080] In the inference process of generative models, there is often a large amount of repetitive input information. In this case, storing intermediate computation results in a storage cluster can avoid redundant calculations of some intermediate results, further reducing inference costs. For example, for the current input information, it is matched with the token corresponding to the intermediate computation result in the storage cluster. If the current input information matches the token corresponding to the intermediate computation result in the storage cluster, the intermediate computation result corresponding to the successfully matched token is retrieved from the storage cluster and used for subsequent calculations. This scheme is a reuse scheme for intermediate computation results, replacing the calculation of intermediate computation results with storage and transmission; it is a storage-based computation solution.

[0081] However, generative models are currently mainly deployed on a single computing node, and these intermediate computation results are only reused within that single computing node. With the widespread deployment of inference services, generative models may be deployed on multiple computing nodes. Therefore, this application provides a method for determining model output results. This method can fully utilize the intermediate computation results stored in different computing nodes, thereby achieving the reuse of intermediate computation results in cross-network (e.g., wide area network) scenarios, realizing the overall scheduling of multiple computing nodes, improving overall scheduling efficiency, thereby increasing the throughput of model inference and reducing inference costs.

[0082] Optionally, different computing nodes can be deployed in different regions or different networks. For example, computing node 1 is deployed in region 1, and computing node 2 is deployed in region 2, with region 1 and region 2 connected via a network. As another example, computing node 1 is deployed in network 1 (e.g., branch network 1), and computing node 2 is deployed in network 2 (e.g., branch network 2), with branch network 1 and branch network 2 connected via a wide area network or a private enterprise network. It should be noted that a computing node can be a single computing device, a computing cluster including multiple computing devices, or a data center including one or more computing devices and one or more network devices, or other forms, which are not limited in this embodiment. The following description uses a data center as an example.

[0083] Finally, the implementation environment involved in the embodiments of this application will be described.

[0084] Please refer to Figure 5 , Figure 5 This is a schematic diagram of an implementation environment provided in an embodiment of this application. The implementation environment includes multiple data centers ( Figure 5 (Illustratively represented by three data centers), each data center includes a control node, a pre-filled cluster, a decoding cluster, and a storage cluster. These data centers are interconnected via wired or wireless connections.

[0085] The generative model is deployed across these multiple data centers, meaning that the same generative model is deployed across these multiple data centers. Each of these data centers can provide inference services through the generative model, and each of these data centers can store intermediate computation results generated during the inference service provision process. Furthermore, the intermediate computation results stored across these multiple data centers can be reused.

[0086] For example, in one of the data centers, the control node can receive user input information, use this input information as input to the generative model, and schedule it to the pre-filled cluster. The pre-filled cluster determines the intermediate computation results corresponding to multiple input tokens included in the input information, and stores these intermediate computation results in the storage cluster. Thus, during the inference process in the decoding cluster, the required intermediate computation results can be retrieved from the storage cluster to determine the output tokens. These output tokens are then used to determine the output results of the generative model for the input information, and the intermediate computation results corresponding to each output token can also be stored in the storage cluster.

[0087] Furthermore, to facilitate the reuse of intermediate computation results across multiple data centers, an index structure can be stored in the storage cluster. This index structure indicates the input and / or output information of the generative model. Subsequently, after obtaining the input information of the generative model, this input information can be matched against the index structures of the multiple data centers to determine if any part of the input information matches any of the index structures. This facilitates the reuse of the intermediate computation results corresponding to the matched information, saving computational costs associated with those intermediate computation results.

[0088] In this embodiment, the control node can be any type of node with control functions; the pre-filled cluster includes multiple pre-filled nodes, which can be any type of node with computing functions; the decoding cluster includes multiple decoding nodes, which can be any type of node with computing functions; and the storage cluster includes multiple storage nodes, which can be any type of node with storage functions. Furthermore, each of the control node, pre-filled node, decoding node, and storage node can be a computer device, a chip, or a module within a computer device. Some or all of these nodes can be integrated into a single computer device, or into multiple different computer devices; this embodiment does not impose any limitations on this.

[0089] Those skilled in the art should understand that the above-described nodes are merely examples, and other existing or future node forms that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0090] Please refer to Figure 6 , Figure 6 This is a schematic diagram of another implementation environment provided in this application embodiment. This implementation environment includes multiple data centers ( Figure 6 (Illustratively representing three data centers) and a scheduling node, each data center including a control node, a pre-filled cluster, a decoding cluster, and a storage cluster. The scheduling node can communicate with each of these data centers individually, via either a wired or wireless connection.

[0091] The generative model is deployed across these multiple data centers, meaning that the same generative model is deployed across these multiple data centers. Each of these data centers can provide inference services through the generative model, and each of these data centers can store intermediate computation results generated during the inference service provision process. Furthermore, the intermediate computation results stored across these multiple data centers can be reused.

[0092] For example, the scheduling node can receive user input information, use it as input to the generative model, and schedule it to one of the multiple data centers. Then, the control node of that data center schedules the input information to a pre-filled cluster. The pre-filled cluster determines the intermediate computation results corresponding to the multiple input tokens included in the input information and stores these intermediate computation results in the storage cluster. Thus, during inference in the decoding cluster, the necessary intermediate computation results can be retrieved from the storage cluster to determine the output tokens. These output tokens are then used to determine the output results of the generative model for that input information, and the intermediate computation results corresponding to each output token can also be stored in the storage cluster.

[0093] Furthermore, to facilitate the reuse of intermediate computation results across multiple data centers, an index structure can be stored in the storage cluster. This index structure indicates the input and / or output information of the generative model. Thus, after the scheduling node subsequently obtains the input information of the generative model, it can match this input information with the index structures of the multiple data centers to determine if any part of the input information matches any of the index structures. This facilitates the reuse of the intermediate computation results corresponding to the matched information, saving computational costs associated with those intermediate computation results.

[0094] In this embodiment, the scheduling node can be any type of node with scheduling functionality, the control node can be any type of node with control functionality, the pre-filled cluster includes multiple pre-filled nodes (which can be any type of node with computing functionality), the decoding cluster includes multiple decoding nodes (which can be any type of node with computing functionality), and the storage cluster includes multiple storage nodes (which can be any type of node with storage functionality). Furthermore, each of the scheduling node, control node, pre-filled node, decoding node, and storage node can be a computer device, a chip, or a module within a computer device. Some or all of these nodes can be integrated into a single computer device, or they can be integrated into multiple different computer devices; this embodiment does not impose any limitations on this.

[0095] Those skilled in the art should understand that the above-described nodes are merely examples, and other existing or future node forms that are applicable to the embodiments of this application should also be included within the scope of protection of the embodiments of this application, and are hereby incorporated by reference.

[0096] It should be noted that the application scenarios and implementation environments described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the emergence of new application scenarios and the evolution of implementation environments, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0097] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a computer device shown in an embodiment of this application. The computer device includes at least one processor 701, a communication bus 702, a memory 703, and at least one communication interface 704.

[0098] The processor 701 can be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0099] The communication bus 702 is used to transmit information between the aforementioned components. The communication bus 702 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0100] The memory 703 may be a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disc (including a compact disc read-only memory (CD-ROM), a compressed optical disc, a laser disc, a digital versatile optical disc, a Blu-ray disc, etc.), a magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 703 may exist independently and be connected to the processor 701 via a communication bus 702. The memory 703 may also be integrated with the processor 701.

[0101] Communication interface 704 uses any transceiver-like device for communicating with other devices or communication networks. Communication interface 704 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0102] As an example, processor 701 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in the diagram.

[0103] As an example, a computer device may include multiple processors, such as Figure 7 The processors 701 and 705 are shown. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0104] In some embodiments, the computer device may further include output devices and input devices. The output device communicates with the processor 701 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 701 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0105] In some embodiments, memory 703 is used to store program code 710 for executing the scheme of this application, and processor 701 can execute the program code 710 stored in memory 703. The program code 710 may include one or more software modules, and the computer device can implement the method described below through processor 701 and program code 710 in memory 703.

[0106] The method for determining the model output results provided in the embodiments of this application will be explained in detail below.

[0107] Please refer to Figure 8 , Figure 8 This is a flowchart illustrating a method for determining model output results provided in an embodiment of this application. This method can be applied to the above... Figure 5 In the implementation environment shown, the subject executing this method is Figure 5 The multiple data centers shown include a first data center, on which the generated model is deployed. The method includes the following steps.

[0108] Step 801: The first data center obtains the current input information of the generative model.

[0109] The current input information of the generative model is the input information for this model inference process. The current input information can be the input information provided by the user. That is, the user terminal obtains the input information provided by the user and sends a request to the first data center. This request carries the input information provided by the user and uses the input information provided by the user as the current input information of the generative model.

[0110] There are various ways for a user terminal to obtain user-provided input information. For example, the user terminal displays an information retrieval interface, retrieves the information provided by the user on the information retrieval interface, and uses the information provided by the user on the information retrieval interface as the user-provided input information. The form of user-provided input information can include various forms, such as text, voice, images, and video.

[0111] The first data center can be any one of the multiple data centers, or it can be the data center closest to the user terminal among the multiple data centers, or it can be the data center with the lightest load among the multiple data centers. The selection of the first data center is not limited in the embodiments of this application.

[0112] The generative model is deployed across multiple data centers, meaning that the same generative model is deployed across these multiple data centers.

[0113] Step 802: The first data center retrieves the index structure of the multiple data centers based on the current input information. The index structure is used to indicate the historical information of the generative model. The historical information includes historical input information and / or historical output results. The multiple data centers store the intermediate calculation results corresponding to the historical information indicated by their respective index structures.

[0114] Historical input information refers to the input information used to generate the model before the current model inference. Historical output results refer to the output results determined by the model generated before the current model inference. Historical input information includes one or more input information entries, and historical output results include one or more output results.

[0115] Because these multiple data centers deploy the same generative model, and each data center can receive input information provided by users to the generative model and use the model to infer output results, each data center has an index structure that indicates the historical information of the generative model. This historical information includes historical input information and / or historical output results. Therefore, for each of these multiple data centers, its index structure indicates the historical information acquired by that data center, which consists of the historical input information acquired by that data center and / or the historical output results determined by that data center through the generative model.

[0116] For example, these multiple data centers include Data Center 1, Data Center 2, and Data Center 3. Historical information includes historical input information and historical output information. Data Center 1 acquires historical input information including input information A, input information B, and input information C. Data Center 1 determines historical output results through a generative model, including output results A, B, and C. Input information A corresponds to output result A, input information B corresponds to output result B, and input information C corresponds to output result C. The index structure of Data Center 1 indicates historical information including input information A, input information B, input information C, output results A, output results B, and output results C. Similarly, Data Center 2 acquires historical input information including input information D and input information E. Data Center 2 determines historical output results through a generative model, including output results D and output results E. Input information D corresponds to output result D, and input information E corresponds to output result E. The index structure of Data Center 2 indicates historical information including input information D, input information E, output results D, and output results E. Data Center 3 acquires historical input information including input information D and input information F. Data Center 3 determines historical output results through a generative model, including output results D and output results F. In this context, input information D corresponds to output result D, and input information F corresponds to output result F. The historical information indicated by the index structure of data center 3 includes input information D, input information F, output result D, and output result F.

[0117] The index structure of a data center can be constructed using historical information acquired by that data center. For example, any piece of information from the historical information acquired by the data center, for ease of description, will be referred to as the target information. The target information can be either input information or output. Based on the tokens included in the target information, and according to the dependencies of these tokens within the target information, a corresponding index structure is generated.

[0118] The tokens included in the target information are obtained by dividing the target information. For example, if the target information is text, it can be segmented into multiple word units, each word unit being a token. If the target information is an image, it can be divided into multiple image blocks, each image block being a token. If the target information is video, each video frame can be divided into multiple image blocks, each image block being a token.

[0119] The method for determining the token described above is one example; in other embodiments, tokens can be determined in other ways. Furthermore, this application does not limit the method of word segmentation for text information, image segmentation, or video frame segmentation.

[0120] For tokens included in the target information, the dependencies between these tokens within the target information are used to indicate the structure of the target information, or in other words, to indicate the position of these tokens within the target information. For example, for text information, if the text information includes multiple tokens, the non-first token depends on each token preceding it. For an image, if the image includes multiple tokens, the non-first token depends on each token preceding it. That is, the non-first token depends on each token preceding it.

[0121] For example, for text information, suppose the text information is a user's question "What color is an apple?" The text information includes tokens "apple", "is", "what", and "color". The token "is" depends on the token "apple", the token "what" depends on the tokens "apple" and "is", and the token "color" depends on the tokens "apple", "is", and "what".

[0122] For example, regarding an image, the image is like... Figure 9 As shown, Figure 9 Includes 6 tokens, according to Figure 9 The order of the tokens is as follows: token 2 depends on token 1, token 3 depends on tokens 1 and 2, token 4 depends on tokens 1, 2 and 3, token 5 depends on tokens 1, 2, 3 and 4, and token 6 depends on tokens 1, 2, 3, 4 and 5.

[0123] In some embodiments, the index structure is a tree structure. If the data center includes a target index structure, then according to the dependency relationships of multiple tokens included in the target information, the tokens included in the target information are added to the target index structure. The target index structure refers to the index structure that includes the first token in the target information. If the data center does not include a target index structure, then based on the tokens included in the target information, a corresponding index structure is generated according to the dependency relationships of these tokens in the target information.

[0124] For example, if the target information is "What color is an apple?", and the target information includes tokens such as "apple", "is", "what", and "color", with "apple" being the first token, then if the index structure stored in the data center includes "apple", then the index structure containing "apple" will be used as the target index structure, and the target index structure will be as follows: Figure 10 As shown. Based on the dependencies of these tokens in the target information, the tokens included in the target information are added to the target index structure, resulting in the following: Figure 11 The index structure is shown below. If the data center does not include the target index structure, the corresponding index structure can be generated directly based on the tokens included in the target information, according to the dependencies of these tokens in the target information, such as... Figure 12 As shown.

[0125] Because the index structure is a tree structure and is used to indicate historical information, and it is built according to the structure of the historical information, with the first token of each historical piece of information being the root node of that historical information in the index structure, for any historical piece of information, if the first token of that historical piece of information exists in the index structure stored in the data center (meaning the data center includes the target index structure), then the root node of that historical piece of information exists in the target index structure, and the token included in that historical piece of information can be directly added to the target index structure. If the first token of that historical piece of information does not exist in the index structure stored in the data center (meaning the data center does not include the target index structure), then the root node of that historical piece of information does not exist in the index structure stored in the data center, and the token included in that historical piece of information cannot be added to the already constructed index structure; the index structure corresponding to that historical piece of information needs to be regenerated. In other words, historical pieces of information with a relationship may correspond to the same index structure, while historical pieces of information without a relationship may correspond to different index structures.

[0126] The above description uses a tree structure as an example. In other embodiments, the index structure can also be other forms of index structure. For example, the index structure can be an information set, where historical information can be directly stored, and a correspondence between input information and output results can be established, along with the tokens included in each piece of information and the dependencies between the tokens included in each piece of information. This application does not limit the form of the index structure.

[0127] In some embodiments, the index structure includes multiple tokens, each token corresponding to an intermediate computation result. Thus, the process of retrieving the index structure of the multiple data centers based on the current input information includes: determining multiple target tokens included in the current input information; and retrieving the index structure of the multiple data centers based on the multiple target tokens and their dependencies in the current input information.

[0128] Based on the above description, for any given piece of information, the dependencies of the tokens included within that information indicate the positions of those tokens within that information. In other words, there are certain dependencies between these tokens. Therefore, to improve retrieval accuracy, the index structure of the multiple data centers can be retrieved using prefix matching, based on these multiple target tokens and their dependencies within the current input information.

[0129] For example, if the current input information is "What season do apples ripen?", the index structure of these multiple data centers would be as follows: Figure 13 As shown ( Figure 13(This is illustrated using two index structures as examples.) The current input information includes multiple target tokens: "apple," "is," "what," "season," and "ripe." Based on the dependencies of these target tokens within the current input information, after retrieving the index structures of these multiple data centers, it is found that... Figure 13 Index structure 1 in the current input includes "apple", "is", and "what", and the dependencies of these tokens in index structure 1 are the same as their dependencies in the current input. Therefore, it can be considered that "what is an apple" in the current input has been matched. Figure 13 Index structure 1 in the database.

[0130] In some embodiments, before retrieving the index structure of the plurality of data centers based on the current input information, the first data center may also obtain the index structure of each of the other data centers besides the first data center.

[0131] For example, the first data center can send index structure retrieval requests to the other data centers at fixed intervals. In response, each of the other data centers can send its own index structure to the first data center, allowing the first data center to retrieve the index structures of the other data centers at fixed intervals. Alternatively, each of the other data centers can directly send its own index structure to the first data center at fixed intervals, allowing the first data center to retrieve the index structures of the other data centers at fixed intervals.

[0132] It should be noted that, regardless of whether the first data center actively retrieves the index structures of other data centers at fixed intervals, or whether other data centers actively send their own index structures to the first data center, the index structures sent by other data centers to the first data center each time can be their own complete index structure or a partial index structure compared to the last update.

[0133] In the above description, the first data center obtains the index structures of the other data centers and retrieves the index structures of these multiple data centers locally based on the current input information. In some other embodiments, the first data center may also send the current input information to the other data centers, so that each of the other data centers can retrieve its own index structure based on the current input information and send its retrieval results to the first data center, thereby enabling the first data center to determine whether the current input information matches the index structures of certain data centers.

[0134] The implementation method of each of the other data centers retrieving their respective index structures based on the current input information is similar to the implementation method of the first data center retrieving the index structures of the multiple data centers based on the current input information, and will not be described again here.

[0135] Step 803: If the search result indicates that part of the information in the current input information matches the index structure of at least one of the multiple data centers, then the first data center determines the current output result of the generation model based on the intermediate calculation results corresponding to the part of information in the at least one data center.

[0136] The at least one data center may or may not include a first data center. In different cases, the method for determining the current output of the generative model based on the intermediate calculation results corresponding to this part of the information in the at least one data center will be different. These will be described separately below.

[0137] In the first implementation, the at least one data center does not include the first data center. In this case, the intermediate calculation results corresponding to the information can be obtained from the at least one data center, and the current output result can be determined by generating a model based on the intermediate calculation results corresponding to the information.

[0138] The number of at least one data center can be equal to or greater than one. Moreover, some of the at least one data centers may not have fully stored the intermediate calculation results corresponding to the information. Under different circumstances, the methods for obtaining the intermediate calculation results corresponding to the information from the at least one data center are different, which will be introduced separately below.

[0139] Case 1: The number of the at least one data center is 1. For example, the at least one data center is a second data center, and the second data center fully stores the intermediate calculation results corresponding to this part of the information. In this case, the intermediate calculation results corresponding to this part of the information can be obtained from the second data center.

[0140] Based on the above description, the index structure includes multiple tokens, each token corresponding to an intermediate computation result, and this information may include one or more tokens. Therefore, the second data center completely storing the intermediate computation results corresponding to this information means that the second data center stores the intermediate computation results corresponding to each token included in this information.

[0141] In some embodiments, multiple communication links exist between the second data center and the first data center. The generated model includes multiple network layers with a sequential inference order. The intermediate computation results corresponding to this part of the information include the intermediate computation results required by the multiple network layers. At this time, based on the inference order of the multiple network layers and the communication quality of the multiple communication links, the intermediate computation results corresponding to this part of the information are obtained from the second data center, so that the intermediate computation results required by the network layer with the earlier inference order are obtained through the communication link with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained through the communication link with low communication quality.

[0142] Since these multiple network layers have a sequential inference order, the intermediate computation results corresponding to this information include the intermediate computation results required by these multiple network layers. Therefore, the intermediate computation results required by the network layers with earlier inference order are obtained through a high-quality communication link, while the intermediate computation results required by the network layers with later inference order are obtained through a low-quality communication link. In this way, the intermediate computation results required by the network layers with earlier inference order can be obtained as quickly as possible. This allows for faster processing of subsequent computations by the network layers with earlier inference order while obtaining the intermediate computation results required by the network layers with later inference order, reducing overall inference latency and thus improving the inference efficiency of the generative model.

[0143] For example, if the current input information is "What season do apples ripen?", and part of the input information matches the index structure of the second data center, that part being "What is an apple?", and the second data center fully stores the intermediate computation results corresponding to that part of the information. Assume the generative model includes three network layers: layer 1, layer 2, and layer 3. The inference order of layer 1 precedes that of layer 2, and the inference order of layer 2 precedes that of layer 3. The intermediate computation result required by layer 1 is the result corresponding to "apple", the intermediate computation result required by layer 2 is the result corresponding to "is", and the intermediate computation result required by layer 3 is the result corresponding to "what". If there are three communication links between the second data center and the first data center, then the intermediate computation result corresponding to "apple" can be transmitted through the communication link with the highest communication quality, the intermediate computation result corresponding to "is" can be transmitted through the communication link with medium communication quality, and the intermediate computation result corresponding to "what" can be transmitted through the communication link with the lowest communication quality.

[0144] For each of these multiple communication links, the communication quality of that link characterizes its data transmission rate, and there is a positive correlation between communication quality and transmission rate. That is, the higher the communication quality of a communication link, the higher its data transmission rate; conversely, the lower the communication quality, the lower its data transmission rate.

[0145] Typically, the communication quality of a communication link can be characterized by various parameters. For example, it can be characterized by parameters such as the latency and bandwidth of the communication link, but this application does not limit this aspect.

[0146] Case 2: The number of at least one data center is greater than 1, and at least one of the at least one data center completely stores the intermediate calculation results corresponding to the information. In this case, select one data center from the at least one data center as the second data center. The second data center completely stores the intermediate calculation results corresponding to the information. Obtain the intermediate calculation results corresponding to the information from the second data center.

[0147] Since the number of at least one data center is greater than 1, meaning that the index structure of at least two data centers hits this part of the information, and among these data centers, some data centers may not have completely stored the intermediate calculation results corresponding to this part of the information, while some data centers may have completely stored the intermediate calculation results corresponding to this part of the information, therefore, the data center that has completely stored the intermediate calculation results corresponding to this part of the information can be selected from the at least one data center, namely the second data center, and then the intermediate calculation results corresponding to this part of the information can be obtained from the second data center.

[0148] If only one of the at least one data center completely stores the intermediate calculation results corresponding to this part of the information, then this data center is directly selected as the second data center. If at least two of the at least one data center completely store the intermediate calculation results corresponding to this part of the information, then one of the at least two data centers is selected as the second data center.

[0149] There are several ways to select one data center from the at least two data centers as the second data center. For example, the data center with the lightest load can be selected as the second data center to achieve a more balanced load across the data centers. Alternatively, the data center closest to the first data center can be selected as the second data center to shorten the time for obtaining intermediate computation results and thus improve inference efficiency.

[0150] The implementation method for obtaining the intermediate calculation results corresponding to this part of the information from the second data center is described in the relevant description in Case 1 above, and will not be repeated here.

[0151] Furthermore, based on the above description, the index structure includes multiple tokens, each token corresponding to an intermediate computation result, and this information may include one or more tokens. Therefore, "the data center does not fully store the intermediate computation results corresponding to this information" means that the data center stores the intermediate computation results corresponding to some of the tokens included in this information, but does not store the intermediate computation results corresponding to all the tokens included in this information.

[0152] Case 3: The number of the at least one data center is greater than 1, and at least two of the at least one data centers completely store the intermediate calculation results corresponding to the information. In this case, the information is taken as the target information, and the intermediate calculation results corresponding to the target information are obtained from the at least two data centers.

[0153] In scenario 2, a data center is selected from at least one data center, and the intermediate calculation result corresponding to that part of the information is obtained from that data center. In scenario 3, the intermediate calculation result corresponding to that part of the information is obtained from at least two data centers, rather than selecting a data center and obtaining the intermediate calculation result corresponding to that part of the information from it.

[0154] In some embodiments, the generative model includes multiple network layers with a sequential inference order, and the intermediate computation results corresponding to the target information include the intermediate computation results required by the multiple network layers. In this case, based on the inference order of the multiple network layers and the communication quality between the first data center and the at least two data centers, the intermediate computation results corresponding to the target information are obtained from the at least two data centers, so that the intermediate computation results required by the network layer with the earlier inference order are obtained from the data center with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained from the data center with low communication quality.

[0155] Since these multiple network layers have a sequential inference order, the intermediate computation results corresponding to this information include the intermediate computation results required by these multiple network layers. Therefore, the intermediate computation results required by the network layers with earlier inference order are obtained through a data center with high communication quality, while the intermediate computation results required by the network layers with later inference order are obtained through a data center with low communication quality. In this way, the intermediate computation results required by the network layers with earlier inference order can be obtained as quickly as possible. This allows for faster subsequent computations to be performed through the network layers with earlier inference order while obtaining the intermediate computation results required by the network layers with later inference order, reducing overall inference latency and improving the inference efficiency of the generative model. In addition, obtaining intermediate computation results from at least two data centers, rather than all from the same data center, can improve the efficiency of obtaining intermediate computation results.

[0156] For example, the current input information is "What season do apples ripen?". Part of the input information matches the index structure of three data centers, specifically "What is an apple?". These three data centers completely store the intermediate computation results corresponding to this part of the information. Assume the generative model includes three network layers: Layer 1, Layer 2, and Layer 3. The inference order of Layer 1 precedes that of Layer 2, and the inference order of Layer 2 precedes that of Layer 3. Layer 1 requires the intermediate computation result corresponding to "apple", Layer 2 requires the intermediate computation result corresponding to "is", and Layer 3 requires the intermediate computation result corresponding to "what". The three data centers are Data Center 1, Data Center 2, and Data Center 3. The communication quality between these three data centers and the first data center, from highest to lowest, is Data Center 1, Data Center 2, and Data Center 3. Thus, the intermediate computation result corresponding to "apple" can be transmitted through Data Center 1, the intermediate computation result corresponding to "is" can be transmitted through Data Center 2, and the intermediate computation result corresponding to "what" can be transmitted through Data Center 3.

[0157] It should be noted that, for any one of the at least two data centers, the detailed implementation process of obtaining intermediate calculation results from that data center can also refer to the content in Case 1 above, and will not be repeated here.

[0158] For each of these multiple data centers, the communication quality of that data center characterizes its data transmission rate, and there is a positive correlation between communication quality and transmission rate. That is, the higher the communication quality of a data center, the higher its data transmission rate; conversely, the lower the communication quality, the lower its data transmission rate.

[0159] Typically, the communication quality of a data center can be characterized by various parameters. For example, it can be characterized by parameters such as latency and bandwidth, but this application does not limit this aspect.

[0160] Case 4: The number of at least one data center is greater than 1. This part of the information includes first information and second information. The second information depends on the first information. At least two of the at least one data centers store the intermediate calculation results corresponding to the first information, and one of the at least one data centers completely stores the intermediate calculation results corresponding to this part of the information. In this case, the first information is used as the target information. The intermediate calculation results corresponding to the target information are obtained from the at least two data centers, and the intermediate calculation results corresponding to the second information are obtained from the one data center.

[0161] The first information includes one or more tokens, and the second information includes one or more tokens. The second information depends on the first information in that each token in the second information depends on each token in the first information.

[0162] The method for obtaining the intermediate calculation results corresponding to the target information from the at least two data centers is described in the relevant description of case 3 above. The method for obtaining the intermediate calculation results corresponding to the second information from one data center is similar to the method for obtaining the intermediate calculation results corresponding to this part of the information in case 1 above. For detailed implementation process, please refer to the relevant description of case 1 above, which will not be repeated here.

[0163] In the second implementation, the at least one data center includes a first data center, and the number of such data centers is greater than one. The information includes first information and second information, where the second information depends on the first information. The first data center does not fully store the intermediate computation results corresponding to this information, but it does store the intermediate computation results corresponding to the first information. In this case, the intermediate computation results corresponding to the second information can be obtained from other data centers besides the first data center. Based on these intermediate computation results, the current output result is determined through a generative model.

[0164] Because there are dependencies between the various pieces of information in this section, and the first data center does not fully store the intermediate computation results corresponding to this section, it is necessary to obtain the intermediate computation results not stored in the first data center from other data centers. The methods for obtaining intermediate computation results from other data centers differ depending on the circumstances, and will be described separately below.

[0165] Case 1: The number of at least one data center is 2, and the at least one data center also includes a second data center, which fully stores the intermediate calculation results corresponding to the information. In this case, the intermediate calculation results corresponding to the second information are obtained from the second data center.

[0166] Since the first data center stores the intermediate calculation results corresponding to the first information but not the intermediate calculation results corresponding to the second information, while the second data center fully stores the intermediate calculation results corresponding to that part of the information, the intermediate calculation results corresponding to the second information can be obtained from the second data center.

[0167] The method of obtaining the intermediate calculation result corresponding to the second information from the second data center is similar to the method of obtaining the intermediate calculation result corresponding to this part of the information from the second data center in the first implementation method above. For detailed implementation process, please refer to the relevant description in the first implementation method above, which will not be repeated here.

[0168] Case 2: The number of at least one data center is greater than 2, and at least two of the at least one data center completely store the intermediate calculation results corresponding to the information. In this case, select one data center from the at least two data centers as the second data center. The second data center completely stores the intermediate calculation results corresponding to the information, and retrieve the intermediate calculation results corresponding to the second information from the second data center.

[0169] The implementation method for selecting the second data center from the at least two data centers is described in the first implementation method above. The implementation method for obtaining the intermediate calculation result corresponding to the second information from the second data center is similar to the method for obtaining the intermediate calculation result corresponding to this part of the information from the second data center in the first implementation method above. For detailed implementation process, please refer to the relevant description in the first implementation method above. It will not be repeated here.

[0170] Case 3: The number of at least one data center is greater than 2, and at least two of the at least one data center completely store the intermediate calculation results corresponding to this part of the information; in this case, the intermediate calculation results corresponding to the second information are obtained from the at least two data centers.

[0171] The method of obtaining the intermediate calculation result corresponding to the second information from the at least two data centers is similar to the method of obtaining the intermediate calculation result corresponding to the information from the at least two data centers in the first implementation method described above. For detailed implementation process, please refer to the relevant description in the first implementation method described above, which will not be repeated here.

[0172] In some embodiments, the first data center may also store the intermediate calculation results corresponding to this part of the information in their entirety. In this case, the current output result can be determined directly based on the intermediate calculation results corresponding to this part of the information in the first data center, without having to obtain them from other data centers, thereby improving inference efficiency.

[0173] At this point, the first data center has obtained all the intermediate computation results corresponding to this part of the information. For the other information in the current input information, the intermediate computation results corresponding to these can be determined. Then, based on the intermediate computation results corresponding to the current input information, the current output result is determined through the generative model.

[0174] In some embodiments, after obtaining intermediate computation results not stored in the first data center from the at least one data center, intermediate computation results obtained from other data centers besides the first data center can be stored, and the index structure of the first data center can be updated. For example, based on the tokens included in this part of the information, the index structure of the first data center is updated according to the dependencies of these tokens in this part of the information.

[0175] In other embodiments, for information other than this part of the current input information, after determining the intermediate calculation results corresponding to these information, the intermediate calculation results corresponding to these information are stored, and the index structure of the first data center is updated. For example, based on the tokens included in these information, the index structure of the first data center is updated according to the dependencies of the tokens included in these information.

[0176] In some other embodiments, for the current output result, the intermediate operation result corresponding to the current output result can be determined, stored, and the index structure of the first data center can be updated. For example, based on the tokens included in the current output result, the index structure of the first data center is updated according to the dependencies of these tokens in the current output result.

[0177] The method for updating the index structure of the first data center is similar to the method for constructing the index structure corresponding to the target information. For details, please refer to the relevant description above, which will not be repeated here.

[0178] After retrieving the index structure of the multiple data centers in step 802 above, if the search results indicate that all information of the current input information matches the index structure of at least one of the multiple data centers, then the current output result is obtained from that at least one data center. In other words, if the historical input information of the generative model is completely identical to the current input information, then the historical output result corresponding to the historical input information can be directly obtained from that at least one data center, and this historical output result can be used as the current output result. This eliminates the need for repeated reasoning processes through the generative model, saving reasoning costs and improving the efficiency of determining the model's output result.

[0179] It's important to note that the generative model is deployed across multiple data centers, which may experience scaling up or down under certain circumstances. For example, if there are many idle resources in these data centers, one data center can be taken offline. In this case, the index results and intermediate computation results stored in the offline data center can be migrated to other data centers. This allows the other data centers to continue performing model inference according to the above methods, reusing intermediate computation results and saving inference costs. Conversely, if resources in these multiple data centers are scarce, one or more new data centers can be added. These new data centers will still deploy the same generative model as the original data centers. For these new data centers, model inference can also be performed according to the above methods to reuse intermediate computation results stored in other data centers, further saving inference costs.

[0180] In this embodiment, based on the current input information of the generative model, the index structure of the multiple data centers is retrieved to determine whether the current input information matches the index structure of the multiple data centers. If the index structure of some data centers is matched, the intermediate computation results corresponding to the matched information can be obtained from these data centers without needing to compute the intermediate computation results corresponding to these information. This is equivalent to replacing the computation process with retrieval, storage, and transmission. Since computation costs are often much higher than transmission costs, this method can save computational costs and thus improve inference efficiency. Furthermore, by reusing the intermediate computation results stored in the multiple data centers across networks (e.g., wide area networks), the overall scheduling of the multiple data centers is achieved, improving overall scheduling efficiency, thereby increasing the throughput of model inference and reducing inference costs. In addition, when obtaining intermediate computation results required by multiple network layers, the intermediate computation results required by the network layer with the earlier inference order can be obtained through a communication link or data center with high communication quality, and the intermediate computation results required by the network layer with the later inference order can be obtained through a communication link or data center with low communication quality. In this way, after obtaining the intermediate computation results required by the network layer with the earlier inference order, subsequent inference operations can be performed without waiting for the intermediate computation results required by all network layers to be obtained, thereby reducing the overall inference latency.

[0181] Please refer to Figure 14 , Figure 14 This is a flowchart illustrating another method for determining model output results provided in an embodiment of this application. This method can be applied to the above... Figure 6 In the implementation environment shown, the subject executing this method is Figure 6 The scheduling node shown generates the model, which is deployed across multiple data centers. The method includes the following steps.

[0182] Step 1401: The scheduling node obtains the current input information of the generated model.

[0183] The process of the scheduling node obtaining the current input information of the generative model is similar to that of the first data center obtaining the current input information of the generative model. For details, please refer to the relevant description in step 801 above, which will not be repeated here.

[0184] Step 1402: The scheduling node retrieves the index structure of the multiple data centers based on the current input information. The index structure is used to indicate the historical information of the generative model. The historical information includes historical input information and / or historical output results. The multiple data centers store the intermediate calculation results corresponding to the historical information indicated by their respective index structures.

[0185] The process of the scheduling node retrieving the index structure of the multiple data centers is similar to the process of the first data center retrieving the index structure of the multiple data centers. For details, please refer to the relevant description in step 802 above, which will not be repeated here.

[0186] In some embodiments, before the scheduling node retrieves the index structure of the multiple data centers based on the current input information, it may also obtain the index structure of the multiple data centers.

[0187] The process of the scheduling node obtaining the index structure of the multiple data centers is similar to the process of the first data center obtaining the index structure of other data centers. For details, please refer to the relevant description in step 802 above, which will not be repeated here.

[0188] Step 1403: If the search result indicates that part of the current input information matches the index structure of at least one of the multiple data centers, the scheduling node determines the current output result of the generation model based on the intermediate calculation result corresponding to the part of information in the at least one data center.

[0189] The scheduling node determines the current output of the generated model based on the intermediate calculation results corresponding to this part of the information in at least one data center in several ways. Several of these methods will be introduced below.

[0190] In the first implementation, a data center is selected from the at least one data center as the second data center. The second data center fully stores the intermediate calculation results corresponding to this part of the information. The current input information is sent to the second data center so that the second data center can determine the current output result based on the intermediate calculation results corresponding to this part of the information through the generation model.

[0191] When the search results indicate that part of the current input information matches the index structure of at least one data center, a second data center that fully stores the intermediate calculation results corresponding to that part of the information can be directly selected from the at least one data center. The current output result is determined through the second data center. There is no need to transmit the intermediate calculation results corresponding to that part of the information. Only the current input information needs to be transmitted to the second data center. Compared with the intermediate calculation results corresponding to that part of the information, the transmission cost and inference cost can be further reduced.

[0192] The process by which the scheduling node selects a second data center from the at least one data center is similar to the process by which the first data center selects a second data center from the at least one data center. For details, please refer to the relevant descriptions above; they will not be repeated here. Similarly, the process by which the second data center determines the current output result based on the intermediate calculation results corresponding to this information is similar to the process by which the first data center determines the current output result based on the intermediate calculation results corresponding to this information. For details, please refer to the relevant descriptions above; they will not be repeated here.

[0193] The second implementation involves determining a third data center from among the multiple data centers and sending the current input information to the third data center. If the at least one data center does not include the third data center, then the intermediate computation results corresponding to this part of the information in the at least one data center are scheduled to the third data center, so that the third data center can determine the current output result based on the intermediate computation results corresponding to this part of the information through a generative model.

[0194] The process of scheduling the intermediate computation results corresponding to this portion of information in at least one data center to the third data center is similar to the process of obtaining the intermediate computation results stored in other data centers described above. Please refer to the relevant descriptions above for details, which will not be repeated here. The process of determining the third data center from among these multiple data centers is similar to the process of selecting the second data center described above. Please refer to the relevant descriptions above for details, which will not be repeated here.

[0195] In some embodiments, if the at least one data center includes a third data center, the third data center does not fully store the intermediate calculation results corresponding to the partial information, and the third data center stores the intermediate calculation results corresponding to the first information, then the intermediate calculation results corresponding to the second information stored in the other data centers besides the third data center in the at least one data center are scheduled to the third data center, so that the third data center can determine the current output result based on the intermediate calculation results corresponding to the partial information by generating a model.

[0196] In the case where at least one data center includes a third data center and the third data center does not fully store the intermediate calculation results corresponding to the information, the intermediate calculation results corresponding to the second information stored in other data centers are scheduled to the third data center, so that the third data center can fully obtain the intermediate calculation results corresponding to the information, and then use the intermediate calculation results corresponding to the information to determine the current output result.

[0197] In this embodiment, based on the current input information of the generative model, the index structure of the multiple data centers is retrieved to determine whether the current input information matches the index structure of the multiple data centers. If the index structure of some data centers is matched, the intermediate computation results corresponding to that part of the information stored in these data centers can be reused without calculating the intermediate computation results corresponding to these information. This is equivalent to replacing the computation process with retrieval, storage, and transmission. Since computation costs are often much higher than transmission costs, this method can save computational costs and thus improve inference efficiency. Furthermore, by reusing the intermediate computation results stored in the multiple data centers across networks (e.g., wide area networks), the overall scheduling of the multiple data centers is achieved, improving overall scheduling efficiency, thereby increasing the throughput of model inference and reducing inference costs. In addition, when obtaining intermediate computation results required by multiple network layers, the intermediate computation results required by the network layer with the earlier inference order can be obtained through a communication link or data center with high communication quality, and the intermediate computation results required by the network layer with the later inference order can be obtained through a communication link or data center with low communication quality. In this way, after obtaining the intermediate computation results required by the network layer with the earlier inference order, subsequent inference operations can be performed without waiting for the intermediate computation results required by all network layers to be obtained, thereby reducing the overall inference latency.

[0198] Figure 15 This is a schematic diagram of a device for determining model output results provided in an embodiment of this application. This device can be implemented as part or all of a computing node by software, hardware, or a combination of both. See also... Figure 15 The device includes: an input information acquisition module 1501, a retrieval module 1502, and an output result determination module 1503.

[0199] The input information acquisition module 1501 is used to acquire the current input information of the generated model;

[0200] The retrieval module 1502 is used to retrieve the index structure of the multiple computing nodes based on the current input information. The index structure is used to indicate the historical information of the generated model. The historical information includes historical input information and / or historical output results. The multiple computing nodes store the intermediate calculation results corresponding to the historical information indicated by their respective index structures.

[0201] The output result determination module 1503 is used to determine the current output result of the generated model based on the intermediate calculation result corresponding to the part of information in the at least one computing node if the retrieval result indicates that part of the current input information hits the index structure of at least one computing node among the multiple computing nodes.

[0202] In one possible implementation, the index structure includes multiple tokens, each token corresponding to an intermediate calculation result; the retrieval module 1502 is specifically used for:

[0203] Determine the multiple target tokens included in the current input information;

[0204] Based on the multiple target tokens, the index structure of the multiple computing nodes is retrieved according to the dependency relationship of the multiple target tokens in the current input information.

[0205] In one possible implementation, the device is included in a first computing node among the plurality of computing nodes; the device further includes:

[0206] The index structure acquisition module is used to acquire the index structure of each computing node other than the first computing node among the multiple computing nodes.

[0207] In one possible implementation, the device includes a first computing node among the plurality of computing nodes, and the at least one computing node does not include the first computing node;

[0208] Output result determination module 1503 includes:

[0209] The result acquisition submodule is used to acquire intermediate calculation results corresponding to partial information from the at least one computing node;

[0210] The output result determination submodule is used to determine the current output result based on the intermediate calculation results corresponding to this part of the information and through the generation model.

[0211] In one possible implementation, the number of the at least one computing node is 1, and it is the second computing node, which fully stores the intermediate calculation results corresponding to this part of the information;

[0212] The submodule for obtaining calculation results is specifically used for:

[0213] Obtain the intermediate calculation results corresponding to this part of the information from the second computing node.

[0214] In one possible implementation, the number of the at least one computing node is greater than 1, and at least one of the at least one computing node fully stores the intermediate calculation results corresponding to the partial information;

[0215] The submodule for obtaining calculation results is specifically used for:

[0216] Select one computing node from the at least one computing node as the second computing node, and the second computing node fully stores the intermediate calculation results corresponding to some information;

[0217] Obtain the intermediate calculation results corresponding to this part of the information from the second computing node.

[0218] In one possible implementation, there are multiple communication links between the second computing node and the first computing node, the generation model includes multiple network layers with a sequential reasoning order, and the intermediate computation results corresponding to this part of the information include the intermediate computation results required by the multiple network layers.

[0219] The submodule for obtaining calculation results is specifically used for:

[0220] Based on the inference order of the multiple network layers and the communication quality of the multiple communication links, the intermediate computation results corresponding to this part of the information are obtained from the second computing node, so that the intermediate computation results required by the network layer with the earlier inference order are obtained through the communication link with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained through the communication link with low communication quality.

[0221] In one possible implementation, the number of the at least one computing node is greater than 1, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information;

[0222] The submodule for obtaining calculation results is specifically used for:

[0223] Using partial information as target information, intermediate computation results corresponding to the target information are obtained from at least two computing nodes.

[0224] In one possible implementation, the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, at least two computing nodes in the at least one computing node store the intermediate operation results corresponding to the first information, and at least one computing node in the at least one computing node fully stores the intermediate operation results corresponding to the partial information.

[0225] The submodule for obtaining calculation results is specifically used for:

[0226] The first information is used as the target information. The intermediate calculation results corresponding to the target information are obtained from the at least two computing nodes, and the intermediate calculation results corresponding to the second information are obtained from one computing node.

[0227] In one possible implementation, the generative model includes multiple network layers with a sequential reasoning order, and the intermediate computation results corresponding to the target information include the intermediate computation results required by the multiple network layers.

[0228] The submodule for obtaining calculation results is specifically used for:

[0229] Based on the inference order of the multiple network layers and the communication quality between the first computing node and at least two computing nodes, intermediate computation results corresponding to the target information are obtained from the at least two computing nodes, so that the intermediate computation results required by the network layer with the earlier inference order are obtained from the computing node with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained from the computing node with low communication quality.

[0230] In one possible implementation, the device includes a first computing node among a plurality of computing nodes, the at least one computing node includes the first computing node, and the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, the first computing node does not completely store the intermediate calculation results corresponding to the partial information, and the first computing node stores the intermediate calculation results corresponding to the first information;

[0231] Output result determination module 1503 includes:

[0232] The calculation result acquisition submodule is used to acquire the intermediate calculation result corresponding to the second information from the other computing nodes besides the first computing node in the at least one computing node;

[0233] The output result determination submodule is used to determine the current output result based on the intermediate calculation results corresponding to this part of the information and through the generation model.

[0234] In one possible implementation, the number of the at least one computing node is 2, and the at least one computing node further includes a second computing node, which fully stores the intermediate calculation results corresponding to the partial information.

[0235] The submodule for obtaining calculation results is specifically used for:

[0236] Obtain the intermediate calculation results corresponding to the second information from the second computing node.

[0237] In one possible implementation, the number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information;

[0238] The submodule for obtaining calculation results is specifically used for:

[0239] Select one computing node from the at least two computing nodes as the second computing node. The second computing node fully stores the intermediate calculation results corresponding to some of the information.

[0240] Obtain the intermediate calculation results corresponding to the second information from the second computing node.

[0241] In one possible implementation, the number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to this part of the information;

[0242] The submodule for obtaining calculation results is specifically used for:

[0243] Obtain the intermediate calculation results corresponding to the second information from at least two computing nodes.

[0244] In one possible implementation, the device further includes:

[0245] A storage module is used to store intermediate computation results obtained from the at least one computing node other than the first computing node, and to update the index structure of the first computing node.

[0246] In one possible implementation, the device is included in the scheduling node; the device further includes:

[0247] The index structure acquisition module is used to obtain the index structure of the multiple computing nodes.

[0248] In one possible implementation, the device is contained within the scheduling node;

[0249] The output result determination module is specifically used for:

[0250] Select one computing node from the at least one computing node as the second computing node, and the second computing node fully stores the intermediate calculation results corresponding to some information;

[0251] The current input information is sent to the second computing node, so that the second computing node can determine the current output result based on the intermediate calculation results corresponding to the partial information and through the generation model.

[0252] In one possible implementation, the device is contained within the scheduling node;

[0253] The output result determination module 1503 is specifically used for:

[0254] Determine the third computing node from among these multiple computing nodes;

[0255] Send the current input information to the third computing node;

[0256] If the at least one computing node does not include a third computing node, then the intermediate computation results corresponding to some information in the at least one computing node are scheduled to the third computing node, so that the third computing node can determine the current output result based on the intermediate computation results corresponding to the partial information by generating a model.

[0257] In one possible implementation, this information includes first information and second information, where the second information depends on the first information; the output result determination module 1503 is further specifically used for:

[0258] If the at least one computing node includes a third computing node, the third computing node does not fully store the intermediate operation results corresponding to the partial information, and the third computing node stores the intermediate operation results corresponding to the first information, then the intermediate operation results corresponding to the second information stored by the other computing nodes in the at least one computing node besides the third computing node are scheduled to the third computing node, so that the third computing node can determine the current output result based on the intermediate operation results corresponding to the partial information through the generation model.

[0259] In one possible implementation, the output determination module 1503 is further used for:

[0260] If the search results indicate that all the information of the current input matches the index structure of at least one of the multiple computing nodes, then the current output result is obtained from that at least one computing node.

[0261] In this embodiment, based on the current input information of the generative model, the index structure of the multiple data centers is retrieved to determine whether the current input information matches the index structure of the multiple data centers. If the index structure of some data centers is matched, the intermediate computation results corresponding to that part of the information stored in these data centers can be reused without calculating the intermediate computation results corresponding to these information. This is equivalent to replacing the computation process with retrieval, storage, and transmission. Since computation costs are often much higher than transmission costs, this method can save computational costs and thus improve inference efficiency. Furthermore, by reusing the intermediate computation results stored in the multiple data centers across networks (e.g., wide area networks), the overall scheduling of the multiple data centers is achieved, improving overall scheduling efficiency, thereby increasing the throughput of model inference and reducing inference costs. In addition, when obtaining intermediate computation results required by multiple network layers, the intermediate computation results required by the network layer with the earlier inference order can be obtained through a communication link or data center with high communication quality, and the intermediate computation results required by the network layer with the later inference order can be obtained through a communication link or data center with low communication quality. In this way, after obtaining the intermediate computation results required by the network layer with the earlier inference order, subsequent inference operations can be performed without waiting for the intermediate computation results required by all network layers to be obtained, thereby reducing the overall inference latency.

[0262] It should be noted that the apparatus for determining model output results provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus for determining model output results and the method for determining model output results provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0263] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method for determining the output result of the model described above.

[0264] This application also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method described above for determining the output result of the model.

[0265] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0266] It should be understood that "multiple" as mentioned herein refers to two or more. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first," "second," etc., are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and the terms "first," "second," etc., do not necessarily imply that they are different.

[0267] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0268] The above descriptions are embodiments provided in this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining the output result of a model, characterized in that, The generated model is deployed on multiple computing nodes; the method includes: Obtain the current input information of the generated model; Based on the current input information, the index structure of the plurality of computing nodes is retrieved. The index structure is used to indicate the historical information of the generative model. The historical information includes historical input information and / or historical output results. The plurality of computing nodes store the intermediate calculation results corresponding to the historical information indicated by their respective index structures. If the search result indicates that a portion of the current input information matches the index structure of at least one of the multiple computing nodes, then the current output result of the generating model is determined based on the intermediate computation results corresponding to the portion of information in the at least one computing node.

2. The method as described in claim 1, characterized in that, The index structure includes multiple tokens, each token corresponding to an intermediate computation result; the step of retrieving the index structure of the multiple computing nodes based on the current input information includes: Determine the multiple target tokens included in the current input information; Based on the multiple target tokens, and according to the dependency relationship of the multiple target tokens in the current input information, the index structure of the multiple computing nodes is retrieved.

3. The method as described in claim 1 or 2, characterized in that, The method is applied to the first computing node among the plurality of computing nodes; Before retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: Obtain the index structure of each computing node other than the first computing node among the plurality of computing nodes.

4. The method according to any one of claims 1-3, characterized in that, The method is applied to a first computing node among the plurality of computing nodes, wherein the at least one computing node does not include the first computing node; Determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: Obtain the intermediate calculation results corresponding to the partial information from the at least one computing node; Based on the intermediate calculation results corresponding to the aforementioned partial information, the current output result is determined through the generation model.

5. The method as described in claim 4, characterized in that, The number of the at least one computing node is 1, and it is the second computing node. The second computing node fully stores the intermediate calculation results corresponding to the partial information. The step of obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: Obtain the intermediate calculation results corresponding to the partial information from the second computing node.

6. The method as described in claim 4, characterized in that, The number of the at least one computing node is greater than 1, and at least one of the at least one computing node completely stores the intermediate calculation results corresponding to the partial information; The step of obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: Select one computing node from the at least one computing node as a second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; Obtain the intermediate calculation results corresponding to the partial information from the second computing node.

7. The method as described in claim 5 or 6, characterized in that, There are multiple communication links between the second computing node and the first computing node. The generation model includes multiple network layers with a sequential reasoning order. The intermediate calculation results corresponding to the partial information include the intermediate calculation results required by the multiple network layers. The step of obtaining the intermediate calculation results corresponding to the partial information from the second computing node includes: Based on the inference order of the multiple network layers and the communication quality of the multiple communication links, intermediate computation results corresponding to the partial information are obtained from the second computing node, so that the intermediate computation results required by the network layer with the earlier inference order are obtained through the communication link with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained through the communication link with low communication quality.

8. The method as described in claim 4, characterized in that, The number of the at least one computing node is greater than 1, and at least two of the at least one computing nodes completely store the intermediate calculation results corresponding to the partial information; The step of obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: Using the aforementioned partial information as target information, intermediate computation results corresponding to the target information are obtained from the at least two computing nodes.

9. The method as described in claim 4, characterized in that, The number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, at least two computing nodes in the at least one computing node store the intermediate calculation results corresponding to the first information, and at least one computing node in the at least one computing node completely stores the intermediate calculation results corresponding to the partial information; The step of obtaining the intermediate calculation results corresponding to the partial information from the at least one computing node includes: Using the first information as target information, intermediate calculation results corresponding to the target information are obtained from the at least two computing nodes, and intermediate calculation results corresponding to the second information are obtained from the one computing node.

10. The method as described in claim 8 or 9, characterized in that, The generative model includes multiple network layers with a sequential reasoning order, and the intermediate computation results corresponding to the target information include the intermediate computation results required by the multiple network layers. The step of obtaining the intermediate computation results corresponding to the target information from the at least two computing nodes includes: Based on the inference order of the multiple network layers and the communication quality between the first computing node and the at least two computing nodes, intermediate computation results corresponding to the target information are obtained from the at least two computing nodes, so that the intermediate computation results required by the network layer with the earlier inference order are obtained from the computing node with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained from the computing node with low communication quality.

11. The method according to any one of claims 1-3, characterized in that, The method is applied to a first computing node among the plurality of computing nodes, wherein the at least one computing node includes the first computing node and the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, the first computing node does not completely store the intermediate operation results corresponding to the partial information, and the first computing node stores the intermediate operation results corresponding to the first information; Determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: Obtain the intermediate calculation results corresponding to the second information from the computing nodes other than the first computing node among the at least one computing node; Based on the intermediate calculation results corresponding to the aforementioned partial information, the current output result is determined through the generation model.

12. The method as described in claim 11, characterized in that, The number of the at least one computing node is 2, and the at least one computing node further includes a second computing node, which fully stores the intermediate calculation results corresponding to the partial information. The step of obtaining the intermediate calculation result corresponding to the second information from the computing nodes other than the first computing node among the at least one computing node includes: Obtain the intermediate calculation result corresponding to the second information from the second computing node.

13. The method as described in claim 11, characterized in that, The number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information; The step of obtaining the intermediate calculation result corresponding to the second information from the computing nodes other than the first computing node among the at least one computing node includes: Select one computing node from the at least two computing nodes as the second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; Obtain the intermediate calculation result corresponding to the second information from the second computing node.

14. The method as described in claim 11, characterized in that, The number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information; The step of obtaining the intermediate calculation result corresponding to the second information from the computing nodes other than the first computing node among the at least one computing node includes: Obtain the intermediate calculation results corresponding to the second information from the at least two computing nodes.

15. The method according to any one of claims 4-14, characterized in that, The method further includes: The intermediate computation results obtained from the at least one computing node other than the first computing node are stored, and the index structure of the first computing node is updated.

16. The method as described in claim 1 or 2, characterized in that, The method is applied to scheduling nodes; Before retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: Obtain the index structure of the multiple computing nodes.

17. The method as described in claim 1, 2, or 16, characterized in that, The method is applied to scheduling nodes; Determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: Select one computing node from the at least one computing node as a second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; The current input information is sent to the second computing node, so that the second computing node determines the current output result based on the intermediate calculation results corresponding to the partial information through the generation model.

18. The method as described in claim 1, 2, or 16, characterized in that, The method is applied to scheduling nodes; Determining the current output result of the generative model based on the intermediate computation results corresponding to the partial information in the at least one computing node includes: A third computing node is determined from the plurality of computing nodes; Send the current input information to the third computing node; If the at least one computing node does not include the third computing node, then the intermediate computation results corresponding to the partial information in the at least one computing node are scheduled to the third computing node, so that the third computing node determines the current output result based on the intermediate computation results corresponding to the partial information through the generation model.

19. The method as described in claim 18, characterized in that, The information includes first information and second information, wherein the second information depends on the first information; After sending the current input information to the third computing node, the method further includes: If the at least one computing node includes the third computing node, the third computing node does not fully store the intermediate operation results corresponding to the partial information, and the third computing node stores the intermediate operation results corresponding to the first information, then the intermediate operation results corresponding to the second information stored by the other computing nodes besides the third computing node in the at least one computing node are scheduled to the third computing node, so that the third computing node determines the current output result based on the intermediate operation results corresponding to the partial information through the generation model.

20. The method according to any one of claims 1-19, characterized in that, After retrieving the index structure of the plurality of computing nodes based on the current input information, the method further includes: If the search result indicates that all the information of the current input information matches the index structure of at least one of the plurality of computing nodes, then the current output result is obtained from the at least one computing node.

21. An apparatus for determining the output result of a model, characterized in that, The generative model is deployed on multiple computing nodes; the device includes: An input information acquisition module is used to acquire the current input information of the generated model; The retrieval module is used to retrieve the index structure of the plurality of computing nodes based on the current input information. The index structure is used to indicate the historical information of the generative model. The historical information includes historical input information and / or historical output results, and the plurality of computing nodes store the intermediate calculation results corresponding to the historical information indicated by their respective index structures. The output result determination module is used to determine the current output result of the generation model based on the intermediate calculation results corresponding to the partial information in the at least one computing node if the retrieval result indicates that a portion of the current input information matches the index structure of at least one computing node among the plurality of computing nodes.

22. The apparatus as claimed in claim 21, characterized in that, The index structure includes multiple tokens, each token corresponding to an intermediate calculation result; the retrieval module is specifically used for: Determine the multiple target tokens included in the current input information; Based on the multiple target tokens, and according to the dependency relationship of the multiple target tokens in the current input information, the index structure of the multiple computing nodes is retrieved.

23. The apparatus as claimed in claim 21 or 22, characterized in that, The device includes a first computing node among the plurality of computing nodes; the device further includes: The index structure acquisition module is used to acquire the index structure of each computing node other than the first computing node among the plurality of computing nodes.

24. The apparatus according to any one of claims 21-23, characterized in that, The device includes a first computing node among the plurality of computing nodes, and the at least one computing node does not include the first computing node; The output result determination module includes: The calculation result acquisition submodule is used to acquire intermediate calculation results corresponding to the partial information from the at least one computing node; The output result determination submodule is used to determine the current output result based on the intermediate calculation results corresponding to the partial information through the generation model.

25. The apparatus as claimed in claim 24, characterized in that, The number of the at least one computing node is 1, and it is the second computing node. The second computing node fully stores the intermediate calculation results corresponding to the partial information. The submodule for obtaining the calculation result is specifically used for: Obtain the intermediate calculation results corresponding to the partial information from the second computing node.

26. The apparatus as claimed in claim 24, characterized in that, The number of the at least one computing node is greater than 1, and at least one of the at least one computing node completely stores the intermediate calculation results corresponding to the partial information; The submodule for obtaining the calculation result is specifically used for: Select one computing node from the at least one computing node as a second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; Obtain the intermediate calculation results corresponding to the partial information from the second computing node.

27. The apparatus as claimed in claim 25 or 26, characterized in that, There are multiple communication links between the second computing node and the first computing node. The generation model includes multiple network layers with a sequential reasoning order. The intermediate calculation results corresponding to the partial information include the intermediate calculation results required by the multiple network layers. The submodule for obtaining the calculation result is specifically used for: Based on the inference order of the multiple network layers and the communication quality of the multiple communication links, intermediate computation results corresponding to the partial information are obtained from the second computing node, so that the intermediate computation results required by the network layer with the earlier inference order are obtained through the communication link with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained through the communication link with low communication quality.

28. The apparatus as claimed in claim 24, characterized in that, The number of the at least one computing node is greater than 1, and at least two of the at least one computing nodes completely store the intermediate calculation results corresponding to the partial information; The submodule for obtaining the calculation result is specifically used for: Using the aforementioned partial information as target information, intermediate computation results corresponding to the target information are obtained from the at least two computing nodes.

29. The apparatus as claimed in claim 24, characterized in that, The number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, at least two computing nodes in the at least one computing node store the intermediate calculation results corresponding to the first information, and at least one computing node in the at least one computing node completely stores the intermediate calculation results corresponding to the partial information; The submodule for obtaining the calculation result is specifically used for: Using the first information as target information, intermediate calculation results corresponding to the target information are obtained from the at least two computing nodes, and intermediate calculation results corresponding to the second information are obtained from the one computing node.

30. The apparatus as claimed in claim 28 or 29, characterized in that, The generative model includes multiple network layers with a sequential reasoning order, and the intermediate computation results corresponding to the target information include the intermediate computation results required by the multiple network layers. The submodule for obtaining the calculation result is specifically used for: Based on the inference order of the multiple network layers and the communication quality between the first computing node and the at least two computing nodes, intermediate computation results corresponding to the target information are obtained from the at least two computing nodes, so that the intermediate computation results required by the network layer with the earlier inference order are obtained from the computing node with high communication quality, and the intermediate computation results required by the network layer with the later inference order are obtained from the computing node with low communication quality.

31. The apparatus according to any one of claims 21-23, characterized in that, The device includes a first computing node among the plurality of computing nodes, the at least one computing node includes the first computing node, and the number of the at least one computing node is greater than 1, the partial information includes first information and second information, the second information depends on the first information, the first computing node does not completely store the intermediate calculation results corresponding to the partial information, and the first computing node stores the intermediate calculation results corresponding to the first information; The output result determination module includes: The calculation result acquisition submodule is used to acquire the intermediate calculation result corresponding to the second information from the other computing nodes besides the first computing node in the at least one computing node; The output result determination submodule is used to determine the current output result based on the intermediate calculation results corresponding to the partial information through the generation model.

32. The apparatus as claimed in claim 31, characterized in that, The number of the at least one computing node is 2, and the at least one computing node further includes a second computing node, which fully stores the intermediate calculation results corresponding to the partial information. The submodule for obtaining the calculation result is specifically used for: Obtain the intermediate calculation result corresponding to the second information from the second computing node.

33. The apparatus as claimed in claim 31, characterized in that, The number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information; The submodule for obtaining the calculation result is specifically used for: Select one computing node from the at least two computing nodes as the second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; Obtain the intermediate calculation result corresponding to the second information from the second computing node.

34. The apparatus as claimed in claim 31, characterized in that, The number of the at least one computing node is greater than 2, and at least two of the at least one computing node completely store the intermediate calculation results corresponding to the partial information; The submodule for obtaining the calculation result is specifically used for: Obtain the intermediate calculation results corresponding to the second information from the at least two computing nodes.

35. The apparatus according to any one of claims 24-34, characterized in that, The device further includes: A storage module is used to store intermediate computation results obtained from other computing nodes besides the first computing node in the at least one computing node, and to update the index structure of the first computing node.

36. The apparatus as claimed in claim 21 or 22, characterized in that, The device is included in the scheduling node; the device further includes: The index structure acquisition module is used to acquire the index structure of the multiple computing nodes.

37. The apparatus as claimed in claim 21, 22 or 36, characterized in that, The device is included in the scheduling node; The output result determination module is specifically used for: Select one computing node from the at least one computing node as a second computing node, and the second computing node fully stores the intermediate calculation results corresponding to the partial information; The current input information is sent to the second computing node, so that the second computing node determines the current output result based on the intermediate calculation results corresponding to the partial information through the generation model.

38. The apparatus as claimed in claim 21, 22 or 36, characterized in that, The device is included in the scheduling node; The output result determination module is specifically used for: A third computing node is determined from the plurality of computing nodes; Send the current input information to the third computing node; If the at least one computing node does not include the third computing node, then the intermediate computation results corresponding to the partial information in the at least one computing node are scheduled to the third computing node, so that the third computing node determines the current output result based on the intermediate computation results corresponding to the partial information through the generation model.

39. The apparatus as claimed in claim 38, characterized in that, The partial information includes first information and second information, wherein the second information depends on the first information; the output result determination module is further specifically used for: If the at least one computing node includes the third computing node, the third computing node does not fully store the intermediate operation results corresponding to the partial information, and the third computing node stores the intermediate operation results corresponding to the first information, then the intermediate operation results corresponding to the second information stored by the other computing nodes besides the third computing node in the at least one computing node are scheduled to the third computing node, so that the third computing node determines the current output result based on the intermediate operation results corresponding to the partial information through the generation model.

40. The apparatus according to any one of claims 21-39, characterized in that, The output result determination module is also used for: If the search result indicates that all the information of the current input information matches the index structure of at least one of the plurality of computing nodes, then the current output result is obtained from the at least one computing node.

41. A computing node, characterized in that, The computing node includes a processor and a memory for storing computer programs, and the processor is configured to execute the computer programs stored in the memory to implement the method of any one of claims 1-20.

42. A computer-readable storage medium, characterized in that, The storage medium stores instructions that, when executed on the computer, cause the computer to perform the method described in any one of claims 1-20.

43. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method according to any one of claims 1-20.

Images