Model inference method, apparatus and computing device

Abstract

Provided in the present application is a model inference method, the method comprising: receiving input information; on the basis of a first decoding algorithm, using a large model to generate output information related to the input information; acquiring the real value of a target index during the process of the large model generating the output information; when the difference value between the real value of the target index and the reference value of the target index corresponding to the first decoding algorithm is greater than a preset threshold value, selecting a second decoding algorithm, the second decoding algorithm being different from the first decoding algorithm; and, on the basis of the second decoding algorithm, using the large model to generate output information related to the input information. The method can increase the throughput of large model inference systems, and reduce the inference latency of large models.

Description

Methods, apparatus, and computing devices for model inference

[0001] This application claims priority to Chinese Patent Application No. 202410788973.X, filed with the China National Intellectual Property Administration on June 19, 2024, entitled "Method, Apparatus and Computing Device for Model Training", and Chinese Patent Application No. 202410954073.8, filed with the China National Intellectual Property Administration on July 16, 2024, entitled "Method, Apparatus and Computing Device for Model Reasoning", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud computing, and more specifically, to a method, apparatus, and computing device for model reasoning. Background Technology

[0003] Large models refer to machine learning models with a large number of parameters and complex computational structures. These models are typically built from deep neural networks and have billions or even hundreds of billions of parameters. A large-model inference system is a system that uses a pre-trained large model to provide inference services. This system receives text input from users or other systems (e.g., prompts from users or other systems), decodes it using the large model, and generates and returns the corresponding text output.

[0004] In a related large-scale model inference system, an autoregressive decoding method is used for inference services. Since autoregressive decoding can only produce one token output at a time, if a request requires multiple outputs, the inference system needs to perform multiple inference processes. This results in low throughput and long latency for the large-scale model inference system.

[0005] Another related large-model inference system accelerates inference through decoding acceleration algorithms. These algorithms are configured once; once selected in deployment, they run throughout the runtime. However, because these algorithms consume additional computational resources, and large-model inference inputs are often large batches, under heavy loads and limited computational resources, they may actually cause the inference system's performance to be worse than autoregressive decoding algorithms. This results in a negative outcome, leading to lower throughput and longer latency for the large-model inference system.

[0006] Therefore, improving the throughput of large-scale model inference systems and reducing the inference latency of large-scale models have become urgent technical problems that need to be solved. Summary of the Invention

[0007] This application provides a model inference method that can improve the throughput of large model inference systems and reduce the inference latency of large models.

[0008] Firstly, a model inference method is provided, applied to a large model inference system, the large model inference system including a large model, the method comprising: receiving input information as input to the large model; generating output information related to the input information using the large model according to a first decoding algorithm; obtaining the true value of a target indicator during the generation of the output information by the large model; selecting a second decoding algorithm if the difference between the true value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, wherein the second decoding algorithm is different from the first decoding algorithm; and generating output information related to the input information using the large model according to the second decoding algorithm.

[0009] In the above technical solution, multiple decoding algorithms can be deployed in advance during the initial deployment stage of the model inference process. The large model inference system can dynamically and adaptively adjust the decoding algorithm used by the large model inference system in real time according to the changes in the indicators that customers are concerned about, so as to achieve stable acceleration of large model inference, reduce latency fluctuations, and improve throughput.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the first decoding algorithm is executed with a higher priority than the second decoding algorithm.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining the first decoding algorithm and the priority order in which the first decoding algorithm is executed based on the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: obtaining the target decoding algorithm set, the target decoding algorithm set including multiple decoding algorithms in priority order, the multiple decoding algorithms including the first decoding algorithm and the second decoding algorithm.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, based on historical input information and the multiple decoding algorithms, the large model is used to generate reference values ​​for the target index corresponding to each of the multiple decoding algorithms; based on the reference values ​​for the target index corresponding to each of the multiple decoding algorithms, the execution order of each of the multiple decoding algorithms is determined; and based on the execution order of each of the multiple decoding algorithms, the target decoding algorithm set is obtained.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the target metric includes any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the large model inference system is deployed in at least one cloud data center managed by a cloud management platform, and each cloud data center is equipped with at least one server.

[0017] Secondly, a model inference apparatus is provided, applied to a large model inference system. The large model inference system includes a large model, and the apparatus includes: a receiving module, an inference module, an acquisition module, and a determination module. The receiving module receives input information as input to the large model. The inference module generates output information related to the input information using the large model according to a first decoding algorithm. The acquisition module acquires the actual value of a target indicator during the generation of the output information by the large model. The determination module selects a second decoding algorithm if the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold. The second decoding algorithm is different from the first decoding algorithm. The inference module also generates output information related to the input information using the large model according to the second decoding algorithm.

[0018] In conjunction with the second aspect, in some implementations of the second aspect, the first decoding algorithm is executed with a higher priority than the second decoding algorithm.

[0019] In conjunction with the second aspect, in some implementations of the second aspect, the determining module is further configured to determine the first decoding algorithm and the priority order in which the first decoding algorithm is executed, based on the reference values ​​of the target indicators corresponding to the first decoding algorithm and the target indicators corresponding to the second decoding algorithm.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition module is further configured to acquire the target decoding algorithm set, which includes multiple decoding algorithms in a priority order, including the first decoding algorithm and the second decoding algorithm.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition module is specifically used to: generate reference values ​​for the target index corresponding to each of the multiple decoding algorithms using the large model based on historical input information and the multiple decoding algorithms; determine the execution order of the multiple decoding algorithms based on the reference values ​​of the target index corresponding to each of the multiple decoding algorithms; and obtain the target decoding algorithm set based on the execution order of the multiple decoding algorithms.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the large model inference system is deployed in at least one cloud data center managed by a cloud management platform, and each cloud data center is equipped with at least one server.

[0025] It should be understood that for the beneficial effects of the second aspect and its various implementations, please refer to the first aspect and its various implementations; they will not be repeated here.

[0026] Thirdly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, causing the program to execute the method of the first aspect or any possible implementation thereof.

[0027] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0028] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method of the first aspect or any possible implementation thereof.

[0029] Fifthly, a chip is provided that acquires and executes instructions to implement the methods described in the first aspect and any implementation thereof.

[0030] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect and any implementation thereof.

[0031] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect and any implementation thereof.

[0032] In a sixth aspect, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first aspect and any implementation thereof.

[0033] In a seventh aspect, a computer program product containing instructions is provided, which, when run by a cluster of computing devices, cause the cluster of computing devices to perform the methods described in the first aspect and any implementation thereof.

[0034] Eighthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first aspect and any implementation thereof.

[0035] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0036] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium.

[0037] A ninth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method as described in the first aspect and any implementation thereof.

[0038] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0039] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0040] Figure 1 is a schematic block diagram of a cloud scenario applicable to an embodiment of this application.

[0041] Figure 2 is a schematic flowchart of a model reasoning method provided in an embodiment of this application.

[0042] Figure 3 is a schematic block diagram of a large model inference system provided in an embodiment of this application.

[0043] Figure 4 is a schematic flowchart of another model reasoning method provided in an embodiment of this application.

[0044] Figure 5 is a schematic diagram of the basic settings interface for large model inference deployment provided in an embodiment of this application.

[0045] Figure 6 is a schematic diagram of an advanced settings interface for large model inference deployment provided in an embodiment of this application.

[0046] Figure 7 is a schematic diagram of the interface of a large model inference system provided in this application, which recommends the sorting of decoding engines corresponding to each batch size according to the speedup ratio.

[0047] Figure 8 is a schematic diagram of a successful large model deployment interface provided in an embodiment of this application.

[0048] Figure 9 is a schematic diagram of monitoring and recording the speedup changes of a large model during inference, according to an embodiment of this application.

[0049] Figure 10 is a schematic block diagram of a model reasoning apparatus 900 provided in an embodiment of this application.

[0050] Figure 11 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application.

[0051] Figure 12 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.

[0052] Figure 13 is a schematic diagram of the connection between computing devices 1500A and 1500B via a network provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0054] This application will present various aspects, embodiments, or features relating to systems comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0055] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0056] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0057] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0058] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0059] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0060] For ease of description, the concepts involved in the embodiments of this application will be explained below.

[0061] 1. Large Model

[0062] Large models refer to machine learning models with a massive number of parameters and complex computational structures. These models are typically built from deep neural networks and have billions or even hundreds of billions of parameters. Large models learn complex patterns and features by training on massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data.

[0063] In this application embodiment, the large model may include, but is not limited to: large language model (LLM), large model using Transformer as the basic model architecture, etc.

[0064] Large Language Models (LLMs) are massive deep learning models pre-trained on massive amounts of text data. They not only generate natural language text but also deeply understand its meaning, handling various natural language tasks such as text summarization, question answering, and translation. These models typically contain billions or even more parameters, learning from vast amounts of text data to predict the next word or generate content related to a given text. Therefore, LLMs and their applications in artificial intelligence have become a global research hotspot, with their remarkable increase in parameter size, jumping from billions to trillions. This increase in parameters allows models to capture the nuances of human language more precisely and understand its complexity more deeply. LLMs have shown significant improvements in absorbing new knowledge, decomposing complex tasks, and aligning text and images. As the technology matures, it will continue to expand its applications, providing more intelligent and personalized services and further improving people's lives and production methods.

[0065] 2. Large-scale model reasoning system

[0066] A large-scale model inference system refers to a system that uses a pre-trained large model to provide inference services. This system receives text input from users or other systems (e.g., prompts from users or other systems), decodes it using the large model, and generates and returns the corresponding text output.

[0067] For example, a large model inference system typically consists of components such as an input processor, a scheduler, a large model engine, and a return processor.

[0068] The aforementioned prompt typically refers to a piece of text or information that serves as an initial input or guide for the large model, instructing it to generate tokens. In other words, the prompt can be seen as the starting point or trigger for the large model to generate tokens; by providing an appropriate prompt, the large model can generate tokens related to the content of that prompt.

[0069] 3. Autoregressive Decoding

[0070] Autoregressive decoding is a decoding strategy used by large models for inference services. Specifically, autoregressive decoding operates on a token-by-token basis. That is, during the autoregressive decoding process, the probability distribution of the next token is calculated based on the existing context (i.e., previously generated tokens), and a token is selected as the next output based on this probability distribution. This process is recursive, meaning that each newly generated token becomes a condition for subsequent predictions. For example, in a text generation task, the first token of the sequence is generated first, then the second token is predicted based on this token, and so on, until a complete sequence is generated.

[0071] It should be understood that the above token can be text, characters, words, image blocks, or other types of data units, depending on the application domain and task of the model.

[0072] 4. Large-scale model inference acceleration technology

[0073] Acceleration techniques for large model inference include, but are not limited to: model compression, decoding methods, underlying optimization, and distributed parallel inference.

[0074] The embodiments of this application mainly focus on acceleration technology for decoding methods. They mainly utilize algorithms that can generate multiple token outputs in a single decoding step to accelerate inference, thereby achieving the goal of accelerating the inference process of large models.

[0075] For example, some commonly used inference acceleration algorithms include, but are not limited to: large-scale model collaborative speculative decoding algorithm, look-ahead decoding algorithm, Medusa decoding algorithm, and Matryoshka speculative decoding algorithm.

[0076] For example, the small-scale model collaborative speculative decoding algorithm pre-generates the outputs of multiple tokens through a small model, and then verifies whether the outputs of these multiple tokens conform to the output distribution of the large model through a single calculation. If they do, the outputs of the multiple tokens from the small model are accepted as the outputs of the large model, thereby achieving the goal of accelerating the inference of the large model.

[0077] For example, lookahead decoding is another acceleration method that does not require small model collaboration. Lookahead decoding adds extra computation in a decoding step to draft and verify possible outputs. Similarly, if multiple tokens are verified, multiple tokens can be generated in a decoding step, which can also achieve the purpose of acceleration.

[0078] In recent years, with the rapid development of large-scale models, intelligent applications built upon them have been widely implemented, influencing how people generate and use information, such as ChatGPT and Sora. A representative characteristic of large-scale models is their massive number of parameters. To continuously improve the application performance of large-scale models, their development is still rapidly expanding in terms of parameter count. In 2023, OpenAI announced ChatGPT with hundreds of billions of parameters, and just a few months later, it announced GPT4 with trillions of parameters. The model's dialogue performance was significantly improved, causing a continuous sensation in the industry. While the massive number of parameters has enabled large-scale models, it has also become a constraint on their widespread adoption. One important reason is the high cost of service-oriented deployment for large-scale model inference. Therefore, how to improve the throughput and reduce the inference latency of large-scale models is a challenging and important issue.

[0079] In a typical large-model inference system, inference services are provided directly based on a pre-trained large model using an autoregressive decoding method. The characteristic of autoregressive decoding is that it can only produce one token output at a time. If a request requires 1024 outputs, the inference system needs to perform 1024 inference processes. Each inference process requires moving the parameters of the large model from the xPU's storage unit to the computation unit, a process that is very slow compared to the computation itself. Therefore, in this large-model inference system, the throughput of the large model during inference is low, and the latency is long.

[0080] Another related large-model inference system attempts to accelerate the inference process of large models through decoding acceleration algorithms. For example, by utilizing additional computing resources, these algorithms can enable the large model to generate multiple token outputs in a single decoding step, thereby reducing the loading and transmission of large model parameters and accelerating the inference process. However, these decoding acceleration algorithms are one-time settings; once selected in the deployment state, they run throughout the runtime. Because these algorithms consume additional computing resources, they can only accelerate large-model inference in very small batches. Since the input to large-model inference is often a large batch, under heavy loads and limited computing resources, these decoding acceleration algorithms may even cause the inference system's performance to be worse than that of autoregressive decoding algorithms, resulting in a negative outcome.

[0081] In summary, the aforementioned large-scale model inference systems can only be configured with one decoding algorithm during initial deployment, and this decoding algorithm will run throughout the entire runtime, resulting in low throughput and long latency for large models during inference.

[0082] In view of this, embodiments of this application provide a method for model inference. This method can support the deployment of multiple decoding algorithms in the initial deployment stage during the model inference process, so that the decoding algorithm used by the large model inference system can be adaptively adjusted according to changes in indicators, thereby improving the throughput of the large model in the inference process and reducing the inference latency of the large model.

[0083] In one possible implementation, the method provided in this application embodiment can be applied to a cloud service scenario, where the method is executed by a cloud management platform within the cloud service scenario. For ease of description, the cloud service scenario will be described in detail below with reference to Figure 1.

[0084] Figure 1 is a schematic block diagram of a cloud scenario applicable to an embodiment of this application. As shown in Figure 1, the cloud scenario may include: a cloud management platform 110, the Internet 120, and a client 130.

[0085] As shown in Figure 1, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources to provide corresponding cloud services to tenants.

[0086] The cloud management platform 110 can be located in a cloud data center and provides access interfaces (such as user interfaces or application program interfaces, APIs). Tenants can use client 130 to remotely access the cloud management platform 110, register a cloud account and password, and log in. After successful authentication of the cloud account and password, the tenant can further select and purchase virtual machines of specific specifications (processor, memory, disk) on the cloud management platform 110. After successful purchase, the cloud management platform 110 provides the remote login account and password for the purchased virtual machine, allowing client 130 to remotely log in and install and run the tenant's applications. Therefore, tenants can create, manage, log in to, and operate virtual machines in the cloud data center through the cloud management platform 110. Virtual machines can also be referred to as Elastic Compute Service (ECS) or Elastic Instances (different cloud service providers may use different names).

[0087] It should be understood that cloud service tenants can be individuals, businesses, schools, hospitals, government agencies, etc.

[0088] The cloud management platform 110 includes, but is not limited to, a user console, compute management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interaction with tenants. The compute management services manage servers running virtual machines and containers, as well as bare metal servers. The network management services manage network services (such as gateways and firewalls). The storage management services manage storage services (such as data bucket services). The authentication services manage tenant account passwords. The image management services manage virtual machine images. Tenants can log in to the cloud management platform 110 via client 130 and the internet 120 to manage their rented cloud services.

[0089] Figure 2 is a schematic flowchart of a model reasoning method provided in an embodiment of this application. As shown in Figure 2, the method may include steps 210-230, which will be described in detail below.

[0090] Step 210: Obtain the set of target decoding algorithms corresponding to the target indicators.

[0091] In this embodiment of the application, the large model inference system can obtain a set of target decoding algorithms corresponding to the target index, which includes multiple decoding algorithms with an execution order.

[0092] The aforementioned target metrics may include, but are not limited to, any of the following metrics: compression ratio, speedup ratio, time to first token (TTFT), and time per output token (TPOT).

[0093] The meaning of each of the above indicators will be explained in detail below.

[0094] 1. Compression ratio

[0095] The current basic decoding algorithm is autoregressive decoding, meaning that each decoding step can only produce one token output. In contrast, large-model decoding acceleration algorithms can produce an average of n (n>=1) tokens per step by consuming some additional computational resources. The ratio of the two decoding algorithms is the compression ratio (cr), where cr = n / 1. It should be understood that a higher compression ratio usually indicates the quality of a decoding acceleration algorithm.

[0096] 2. Acceleration Ratio

[0097] Assume that the time consumed by each decoding step of the autoregressive decoding is t. r The time consumed by each decoding step of a certain decoding acceleration algorithm is t. e The speedup ratio is calculated as follows: When the speedup ratio is equal to 1, it means that the decoding speedup algorithm does not bring any speedup benefits; when the speedup ratio is less than 1, it means that the decoding speedup algorithm runs less efficiently than the autoregressive decoding algorithm; when the speedup ratio is greater than 1, it means that the decoding speedup algorithm can achieve the purpose of speeding up inference.

[0098] 3. TTFT

[0099] TTFT refers to the latency of the first token. That is, the delay from input to output of the first token. For example, in online streaming applications, TTFT is the most important metric because it determines the user experience.

[0100] 4. TPOT

[0101] TPOT refers to the latency of each output token (excluding the first token). For example, in offline batch processing applications, TPOT is the most important metric because it determines the time of the entire inference process.

[0102] For example, the aforementioned multiple decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm. This decoding acceleration algorithm includes, but is not limited to: a large-scale model collaborative speculative decoding algorithm, a look-ahead decoding algorithm, a Medusa decoding algorithm, and a Matryoshka speculative decoding algorithm.

[0103] In this application embodiment, there are multiple ways to obtain the target decoding algorithm set corresponding to the target index in the large model inference system. This application embodiment does not specifically limit this. Two possible implementation methods are introduced below.

[0104] Implementation Method 1: The large-scale model inference system obtains a first set of decoding algorithms and recommends it to the client. If the client does not modify the execution order of the various decoding algorithms in this first set, the client will send a confirmation instruction to the large-scale model inference system. Based on the client's confirmation instruction, the large-scale model inference system can use the first set of decoding algorithms as the target set of decoding algorithms.

[0105] Implementation Method 2: The large model inference system obtains a first set of decoding algorithms and recommends it to the client. If the client modifies the execution order of multiple decoding algorithms in the first set based on their own experience, the client will send a second set of decoding algorithms to the large model inference system. This second set is the set obtained after the client modified the execution order of the multiple decoding algorithms included in the first set. After receiving the second set of decoding algorithms sent by the client, the large model inference system can use it as the target set of decoding algorithms.

[0106] The process of obtaining the first set of decoding algorithms for the large model inference system is described in detail below.

[0107] As an example, a large-scale model inference system can be piloted to obtain reference values ​​for the target metrics corresponding to multiple decoding algorithms. Based on these reference values, the execution order of the decoding algorithms is determined, resulting in a first set of decoding algorithms, which includes multiple decoding algorithms with an execution order.

[0108] For example, taking speedup as the target metric, multiple decoding algorithms include: Decoding Algorithm 1, Decoding Algorithm 2, Decoding Algorithm 3, and Decoding Algorithm 4. Assume the speedup for Decoding Algorithm 1 is 3.0, for Decoding Algorithm 2 it is 2.0, for Decoding Algorithm 3 it is 2.5, and for Decoding Algorithm 4 it is 1.0. The execution order or priority of these decoding algorithms, from highest to lowest, is: Decoding Algorithm 1, Decoding Algorithm 3, Decoding Algorithm 2, and Decoding Algorithm 4.

[0109] Step 220: Based on the values ​​of the target indicators obtained by the large model during the actual inference process, and the reference values ​​of the target indicators corresponding to multiple decoding algorithms, adaptively select the target decoding algorithm from multiple decoding algorithms.

[0110] In this embodiment, the large model inference system can collect the values ​​of target indicators in real time during the actual inference process of the large model. Based on the collected target indicator values ​​and the reference values ​​of the target indicators corresponding to multiple decoding algorithms, the system adaptively selects the target decoding algorithm from among multiple decoding algorithms.

[0111] As an example, the highest priority first decoding algorithm in the set of target decoding algorithms can be used as the target decoding algorithm mentioned above.

[0112] In another example, if the large model inference system is currently using the first decoding algorithm, it can also collect the value of the target metric in real time. Under certain switching conditions, the large model inference system will switch the decoding algorithm from the first to the second. That is, the second decoding algorithm from the target decoding algorithm set can be used as the target decoding algorithm.

[0113] As an example, the switching condition mentioned above could be that the first difference is less than a preset threshold. That is, if the first difference is less than the preset threshold, the decoding algorithm can be switched from the first decoding algorithm to the second decoding algorithm. Here, the first difference is the difference between the reference value of the target indicator corresponding to the first decoding algorithm and the value of the target indicator obtained by the large model during inference.

[0114] Step 230: Based on the input information of the large model and the target decoding algorithm, use the large model to obtain the output information of the large model.

[0115] In this embodiment of the application, the large model inference system can receive input information and, based on the input information of the large model and the target decoding algorithm, obtain the output information of the large model.

[0116] In the above technical solution, multiple decoding algorithms can be deployed in advance during the initial deployment stage of the model inference process. The large model inference system can dynamically and adaptively adjust the decoding algorithm used by the large model inference system in real time according to the changes in the indicators that customers are concerned about, so as to achieve stable acceleration of large model inference, reduce latency fluctuations, and improve throughput.

[0117] Figure 3 is a schematic block diagram of a large model inference system provided in an embodiment of this application. As exemplarily shown in Figure 3, the system may include: an input receiving unit, a scheduler, a large model engine, a decoding engine monitoring unit, a decoding engine switching unit, multiple decoding engines (e.g., decoding engine 1, decoding engine 2, decoding engine 3), a draft pool, an output return unit, etc.

[0118] For example, the input receiving unit described above is used to receive input requests in the running state and transmit the received requests to the scheduler. The scheduler transmits the request to the large model engine according to the current resource usage. The large model engine can perform inference according to the currently set decoding engine 1 (which executes the currently set decoding algorithm 1).

[0119] During the aforementioned inference process, the decoding engine monitoring unit acquires changes in various metrics and determines whether the decoding algorithm needs adjustment, i.e., whether the currently set decoding engine 1 needs to be switched. For example, using the metric as the speedup ratio, if the current speedup ratio is greater than or equal to the predetermined speedup ratio of decoding engine 1, then the current decoding engine 1 will continue to be used for inference. If the current speedup ratio is less than the predetermined speedup ratio of decoding engine 1, the decoding engine switching unit switches to decoding engine 2, which consumes fewer resources, with the native autoregressive decoding engine serving as the final backup solution.

[0120] After the above reasoning process is completed, the reasoning system can return the final output to the user through the output return unit.

[0121] The specific implementation process of a model reasoning method provided in this application embodiment will be described in detail below with reference to Figure 4. It should be understood that the examples in Figure 4 are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to the specific values ​​or specific scenarios illustrated in Figure 4. Those skilled in the art can obviously make various equivalent modifications or changes based on the examples given below in Figure 4, and such modifications and changes also fall within the scope of the embodiments of this application.

[0122] Figure 4 is a schematic flowchart of another model reasoning method provided in an embodiment of this application. As shown in Figure 4, the method may include steps 410-440, which will be described in detail below.

[0123] Step 410: The large model inference system obtains the large model parameters used by the customer when deploying the large model inference service.

[0124] In this embodiment of the application, when deploying a large model inference service, the customer can submit the large model that needs to be run or used, for example, submit the parameters of the large model that needs to be run or used to the large model inference system.

[0125] As an example, the parameters of the large model submitted by the above customer to the large model inference system can come from local storage or from an online storage system, such as object storage service (OBS).

[0126] Optionally, the above customers can also set some other basic parameters of the large model, such as: the maximum inference length supported by the large model, whether the large model uses fp16 precision, etc.

[0127] For example, Figure 5 is a schematic diagram of the basic settings interface for deploying a large model inference system. As shown in Figure 4, customers can upload large model parameters and some other basic parameters of the large model through this interface.

[0128] For example, the parameters of a large model come from an online storage system. The customer can enter "obs: / / url" in the first dialog box of the interface shown in Figure 5 and click the "Upload Model Parameters" button. The large model inference system can then obtain the parameters of the large model from OBS based on the Uniform Resource Locator (URL).

[0129] For example, customers can also set the maximum inference length supported by the large model to "4098" and whether the large model uses fp16 precision to "Yes" in the interface shown in Figure 5.

[0130] Step 420: The large model inference system obtains other advanced settings for the large model inference service from the client.

[0131] In this embodiment of the application, when deploying the large model inference service, the customer can also make some other advanced settings. For example, the customer can set some decoding methods used by the large model when performing inference. These decoding methods include basic decoding methods and multiple decoding acceleration methods.

[0132] As an example, the above decoding acceleration methods may include, but are not limited to: big-small model speculative decoding algorithm, look-ahead decoding algorithm, Medusa decoding algorithm, and Matryoshka speculative decoding algorithm.

[0133] For example, Figure 6 is a schematic diagram of an advanced settings interface for deploying a large model inference system. As shown in Figure 6, customers can use this interface to set various decoding algorithms used by the large model during inference.

[0134] For example, as shown in Figure 6, the customer sets the basic decoding algorithm used by the large model during inference to an autoregressive decoding algorithm. Exemplarily, as shown in Figure 6, the autoregressive decoding engine is used to execute the autoregressive decoding algorithm.

[0135] For example, a customer might set the decoding acceleration algorithm used by the large model during inference to a speculative decoding algorithm for both large and small models, a look-ahead decoding algorithm, and a Medusa decoding algorithm. For instance, as shown in Figure 6, the speculative decoding engine for the large and small models executes the speculative decoding algorithm, the look-ahead decoding engine executes the look-ahead decoding algorithm, and the Medusa decoding engine executes the Medusa decoding algorithm.

[0136] It should be noted that, depending on the different decoding acceleration methods, customers also need to upload or provide relevant supporting files.

[0137] For example, for the big-small model speculative decoding algorithm, the customer also needs to upload the parameters of the small model used in the algorithm through the interface shown in Figure 6. For instance, assuming the parameters of the small model come from an online storage system, the customer can enter "obs: / / url" in the interface shown in Figure 5 and click the "Upload Small Model" button. The big model inference system can then retrieve the parameters of the small model used in the big-small model speculative decoding algorithm from OBS based on the URL.

[0138] For example, regarding the Medusa decoding algorithm, customers also need to upload some additional parameters used in the algorithm through the interface shown in Figure 6. For instance, assuming these additional parameters come from an online storage system, customers can enter "obs: / / url" in the interface shown in Figure 5 and click the "Upload Additional Model" button. The large model inference system can then retrieve these additional parameters from OBS based on the URL.

[0139] It should be understood that, for the look-ahead decoding algorithm, since the large model inference system does not require related supporting files when executing the algorithm, customers do not need to upload the look-ahead decoding algorithm-related files on the interface shown in Figure 6.

[0140] Step 430: The large model inference system obtains the ranking of various decoding engines corresponding to each batch size under the target metrics of the final deployment.

[0141] In this embodiment, after the customer completes the above configuration, the large model inference system can run for a period of time, and obtain various metrics for different decoding engines under different batch size settings. The system then displays to the customer the optimal ranking of various decoding engines for each batch size under different metrics for their reference.

[0142] As an example, the above metrics may include, but are not limited to: compression ratio, speedup ratio, time to first token (TTFT), and time per output token (TPOT).

[0143] In one possible implementation, the customer can select the target metrics they are interested in from various indicators, and then manually intervene or adjust the optimal ranking of multiple decoding engines corresponding to each batch size recommended by the large model inference system under the target metrics they are interested in, to form the final deployment plan.

[0144] For example, if the customer does not adjust the optimal ranking of the various decoding engines corresponding to each batch size recommended by the large model inference system, then the optimal ranking of the various decoding engines corresponding to each batch size recommended by the large model inference system can be used as the final deployment solution.

[0145] For example, if the customer adjusts the optimal ranking of the various decoding engines corresponding to each batch size recommended by the large model inference system, and obtains the final ranking of the various decoding engines corresponding to each batch size, then the final ranking of the various decoding engines corresponding to each batch size can be used as the final deployment scheme.

[0146] For example, taking the speedup ratio as the target metric, Figure 7 shows the ranking of decoding engines corresponding to each batch size recommended by the large model inference system based on the speedup ratio.

[0147] For example, as shown in Figure 7, for a batch size of 1-4 prompts, the speedup ratio for accelerated decoding engine 3 is 3.0, for accelerated decoding engine 2 it is 2.5, for accelerated decoding engine 1 it is 2.0, and for the autoregressive decoding engine it is 1.0. Therefore, based on the speedup ratio of each decoding engine, for a batch size of 1-4 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 3, accelerated decoding engine 2, accelerated decoding engine 1, and autoregressive decoding engine.

[0148] For example, as shown in Figure 7, for a batch size of 5-12 prompts, the speedup ratio for accelerated decoding engine 3 is 2.5, for accelerated decoding engine 2 it is 2.2, for accelerated decoding engine 1 it is 1.8, and for the autoregressive decoding engine it is 1.0. Therefore, based on the speedup ratios of each decoding engine, for this batch size of 5-12 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 3, accelerated decoding engine 2, accelerated decoding engine 1, and autoregressive decoding engine.

[0149] For example, as shown in Figure 7, for a batch size of 13-32 prompts, the speedup ratio for accelerated decoding engine 2 is 1.8, for accelerated decoding engine 3 it is 1.5, for accelerated decoding engine 1 it is 1.3, and for the autoregressive decoding engine it is 1.0. Therefore, based on the speedup ratios of each decoding engine, for this batch size of 13-32 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 2, accelerated decoding engine 3, accelerated decoding engine 1, and autoregressive decoding engine.

[0150] For example, as shown in Figure 7, for a batch size with more than 33 prompts, the speedup of the autoregressive decoding engine is 1.0, the speedup of accelerated decoding engine 1 is 0.9, the speedup of accelerated decoding engine 2 is 0.8, and the speedup of accelerated decoding engine 3 is 0.8. Therefore, based on the speedup of each decoding engine, for a batch size with more than 33 prompts, the recommended decoding engines for the large model inference system are, in order: autoregressive decoding engine, accelerated decoding engine 1, accelerated decoding engine 2, and accelerated decoding engine 3.

[0151] It should be noted that the autoregressive decoding engine described above executes an autoregressive decoding algorithm, while accelerated decoding engines 1, 2, and 3 execute different decoding acceleration algorithms. For a detailed description of the decoding acceleration algorithms, please refer to the explanation above; it will not be repeated here.

[0152] As shown in Figure 7, the interface also includes "Manual Settings," which allows customers to manually intervene or adjust the optimal ranking of various decoding engines corresponding to each batch size recommended by the large model inference system, thus forming the final deployment plan.

[0153] In this embodiment of the application, taking the speedup ratio as an example, the large model inference system can also adaptively switch the current decoding engine to a more suitable decoding engine based on the change in the speedup ratio of each decoding engine in its running state and the switching threshold.

[0154] For example, as shown in Figure 7, users can use the switching threshold set automatically by the system, or they can set the switching threshold manually. This application embodiment does not specifically limit this.

[0155] In this embodiment of the application, after the above operations are completed, the large model can be considered successfully deployed, and the large model inference system can provide the client with the interface shown in Figure 8. As an example, the interface shown in Figure 8 may include an indication of successful model deployment (e.g., 100%), and provide an API (e.g., https: / / service-api) for the client to call.

[0156] Step 440: The large model inference system adaptively switches the current decoding engine to a more suitable decoding engine based on the change of the target index of the large model during inference and the switching threshold.

[0157] In this embodiment of the application, when the large model is in the inference running state, the large model inference system can also determine the batch size corresponding to the target index based on the number of prompts in the inference input, and use the decoding engine with the highest priority in the decoding engine ranking corresponding to the batch size.

[0158] The aforementioned large model inference system can also monitor changes in the target metrics of the large model during inference in real time, and adaptively switch the current decoding engine to a more suitable decoding engine based on the switching threshold according to the changes in the target metrics. For example, taking the speedup ratio as an example, as shown in Figure 9, the speedup ratio monitoring unit in the large model inference system can monitor and record changes in the target metrics of the large model during inference in real time.

[0159] For example, as shown in Figure 9, when the speedup ratio is consistently at a high level, the large model inference system can maintain the currently used decoding engine. However, when computational resources are scarce, the load is heavy, and the speedup ratio is significantly decreasing, it needs to switch to a decoding engine that uses fewer resources and has a lower speedup ratio, until it switches to an autoregressive decoding engine, thereby ensuring stable acceleration of the large model inference system.

[0160] For example, taking the speedup ratio as the target metric and the batch size range of the input prompt as 5-12, the customer deploys the following decoding engines in sequence: Accelerated Decoding Engine 3, Accelerated Decoding Engine 2, Accelerated Decoding Engine 1, and Autoregressive Decoding Engine. For instance, when the large model is in inference mode, the large model inference system prioritizes using Accelerated Decoding Engine 3 to execute the corresponding decoding acceleration algorithm to generate the output information. However, if the large model inference system detects a significant decrease in the speedup ratio during inference services, it needs to switch from Accelerated Decoding Engine 3 to Accelerated Decoding Engine 2, which then executes the corresponding decoding acceleration algorithm to generate the output information. Similarly, if the large model inference system detects a continued significant decrease in the speedup ratio during inference services, it needs to switch from Accelerated Decoding Engine 2 to Accelerated Decoding Engine 1, and so on, until it switches to the autoregressive decoding engine.

[0161] In this embodiment of the application, during the switching process of the large model inference system, system maintenance personnel may decide whether to switch, or the large model inference system may switch automatically according to the set switching threshold.

[0162] As an example, during the switching process, the large model inference system first sends a notification to the system maintenance personnel. If the system maintenance personnel make a decision within a given time, their action takes precedence. Otherwise, the large model inference system automatically switches the decoding engine based on the set switching threshold.

[0163] In the above technical solution, the decoding engine monitoring unit can monitor changes in various indicators such as speedup ratio when the system is running at full resource capacity, prevent the deterioration of the inference system's acceleration effect, stabilize the inference acceleration effect to the greatest extent, and ensure high throughput and low latency.

[0164] The methods provided by the embodiments of this application have been described in detail above with reference to Figures 1 to 9. The embodiments of the apparatus of this application will be described in detail below with reference to Figures 10 to 13. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the preceding method embodiments.

[0165] Figure 10 is a schematic block diagram of a model inference apparatus 1000 provided in an embodiment of this application. The apparatus 1000 can be implemented by software, hardware, or a combination of both. The apparatus 1000 provided in this embodiment can implement the method flow shown in Figure 2 or Figure 4 of this embodiment. The apparatus 1000 includes: a receiving module 1010, an inference module 1020, an acquisition module 1030, and a determination module 1040. The receiving module 1010 receives input information as input to the large model; the inference module 1020 generates output information related to the input information using the large model according to a first decoding algorithm; the acquisition module 1030 acquires the true value of the target indicator during the generation of the output information by the large model; the determination module 1040 selects a second decoding algorithm when the difference between the true value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, wherein the second decoding algorithm is different from the first decoding algorithm; the inference module also generates output information related to the input information using the large model according to the second decoding algorithm.

[0166] Optionally, the first decoding algorithm is executed with a higher priority than the second decoding algorithm.

[0167] Optionally, the determining module 1040 is further configured to determine the first decoding algorithm and the priority order in which the first decoding algorithm is executed, based on the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

[0168] Optionally, the acquisition module 1030 is further configured to acquire the target decoding algorithm set, which includes multiple decoding algorithms with priority order, including the first decoding algorithm and the second decoding algorithm.

[0169] Optionally, the acquisition module 1030 is specifically used to: generate reference values ​​for the target index corresponding to each of the multiple decoding algorithms using the large model based on historical input information and the multiple decoding algorithms; determine the execution order of the multiple decoding algorithms based on the reference values ​​of the target index corresponding to each of the multiple decoding algorithms; and obtain the target decoding algorithm set based on the execution order of the multiple decoding algorithms.

[0170] Optional target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

[0171] Optionally, the plurality of decoding algorithms includes an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0172] Optionally, the large model inference system is deployed in at least one cloud data center managed by the cloud management platform, and each cloud data center has at least one server.

[0173] The device 1000 here can be embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0174] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. For instance, the implementation of receiving module 1010 will be described below. Similarly, the implementation of other modules, such as inference module 1020, acquisition module 1030, determination module 1040, and recommendation module, can refer to the implementation of receiving module 1010.

[0175] As an example of a software functional unit, the receiving module 1010 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the receiving module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0176] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0177] As an example of a hardware functional unit, the receiving module 1010 may include at least one computing device, such as a server. Alternatively, the receiving module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0178] The multiple computing devices included in the receiving module 1010 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the receiving module 1010 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the receiving module 1010 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0179] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] It should be noted that the above embodiments of the device, when executing the above methods, are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the receiving module 1010 can be used to execute any step in the above methods, the inference module 1020 can be used to execute any step in the above methods, the acquisition module 1030 can be used to execute any step in the above methods, the determination module 1040 can be used to execute any step in the above methods, and the recommendation module can be used to execute any step in the above methods. The steps implemented by the receiving module 1010, the inference module 1020, the acquisition module 1030, the determination module 1040, and the recommendation module can be specified as needed. By implementing different steps in the above methods through the receiving module 1010, the inference module 1020, the acquisition module 1030, the determination module 1040, and the recommendation module, all the functions of the above device can be realized.

[0181] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0182] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0183] The following describes in detail, with reference to Figure 11, a computing device provided in an embodiment of this application.

[0184] Figure 11 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of this application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in Figure 11 includes at least one processor 1510 and a memory 1520.

[0185] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0186] The processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the method provided in this application. Alternatively, the processor 1510 executes instructions in the memory 1520, causing the computing device 1500 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0187] Optionally, the computing device 1500 also includes a communication interface 1530. The communication interface 1530 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1500 and other devices or communication networks.

[0188] Optionally, the computing device 1500 also includes a system bus 1540, wherein the processor 1510, memory 1520, and communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540; for example, the processor 1510 can perform data read / write or code execution in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 11, but this does not mean that there is only one bus or one type of bus.

[0189] In one possible implementation, the processor 1510 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1520 or cache 1516.

[0190] Optionally, processor 1510 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1510 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, processor 1510 may be a central processing unit (CPU).

[0191] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0192] Optionally, the processing unit 1512, also known as the core, is the most important component of the processor. The processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0193] In one implementation example, the memory control unit 1514 controls the data interaction between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0194] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbitrator (not shown in Figure 11) is configured on the system bus to handle and coordinate contention for access by the multiple processing units 1512.

[0195] In one implementation example, the processing unit 1512 and the memory control unit 1514 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1512 and the memory control unit 1514.

[0196] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When the processing unit 1512 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, its purpose is to help the processing unit 1512 run faster.

[0197] The memory 1520 provides runtime space for processes in the computing device 1500. For example, the memory 1520 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1520. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1520 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0198] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 1510, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1510 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1512 will send the result out.

[0199] By way of example and not limitation, memory 1520 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0200] The structure of the computing device 1500 listed above is merely illustrative and is not limited thereto. The computing device 1500 in this application includes various hardware components in existing computer systems. For example, the computing device 1500 also includes other memories besides memory 1520, such as disk storage. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1500 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 1500 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in FIG11.

[0201] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0202] As shown in Figure 12, the computing device cluster includes at least one computing device 1500. The memory 1520 of one or more computing devices 1500 in the computing device cluster may store the same instructions for performing the methods described above.

[0203] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also each store a portion of the instructions for executing the above-described methods. In other words, a combination of one or more computing devices 1500 can jointly execute the instructions of the above-described methods.

[0204] It should be noted that the memory 1520 in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1520 of different computing devices 1500 can implement the functions of one or more modules within the aforementioned device.

[0205] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. As shown in Figure 13, two computing devices, 1500A and 1500B, are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.

[0206] It should be understood that the functions of computing device 1500A shown in Figure 13 can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0207] In this embodiment, a computer program product containing instructions is also provided. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0208] In this embodiment, a computer-readable storage medium is also provided. This computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method described above.

[0209] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0210] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0211] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0215] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for model reasoning, characterized in that, The method is applied to a large-model inference system, wherein the large-model inference system includes a large model, and the method includes: Receive input information, which serves as the input to the large model; According to the first decoding algorithm, output information related to the input information is generated using the large model; Obtain the actual values ​​of the target indicators during the process of generating the output information by the large model; If the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, a second decoding algorithm is selected, wherein the second decoding algorithm is different from the first decoding algorithm. According to the second decoding algorithm, output information related to the input information is generated using the large model.

2. The method according to claim 1, characterized in that, The first decoding algorithm is executed with a higher priority than the second decoding algorithm.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the reference values ​​of the target indicators corresponding to the first decoding algorithm and the target indicators corresponding to the second decoding algorithm, the first decoding algorithm and the priority order in which the first decoding algorithm is executed are determined.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a target decoding algorithm set, the target decoding algorithm set including multiple decoding algorithms with priority order, the multiple decoding algorithms including the first decoding algorithm and the second decoding algorithm.

5. The method according to claim 4, characterized in that, The set of target decoding algorithms includes: Based on historical input information and the multiple decoding algorithms, the large model is used to generate reference values ​​for the target indicators corresponding to each of the multiple decoding algorithms. Based on the reference values ​​of the target indicators corresponding to the multiple decoding algorithms, the execution order of each of the multiple decoding algorithms is determined. The target decoding algorithm set is obtained according to the order in which the multiple decoding algorithms are executed.

6. The method according to any one of claims 1 to 5, characterized in that, The target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

7. The method according to any one of claims 4 to 6, characterized in that, The plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

8. The method according to any one of claims 1 to 7, characterized in that, The large model inference system is deployed in at least one cloud data center managed by the cloud management platform, and each cloud data center is equipped with at least one server.

9. A device for model reasoning, characterized in that, The device is applied to a large-model inference system, the large-model inference system including a large model, and the device includes: A receiving module is used to receive input information, which serves as the input to the large model; The inference module is used to generate output information related to the input information using the large model based on the first decoding algorithm; The acquisition module is used to acquire the actual values ​​of the target indicators during the process of generating the output information by the large model; The determining module is used to select a second decoding algorithm when the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, wherein the second decoding algorithm is different from the first decoding algorithm; The reasoning module is further configured to generate output information related to the input information using the large model based on the second decoding algorithm.

10. The apparatus according to claim 9, characterized in that, The first decoding algorithm is executed with a higher priority than the second decoding algorithm.

11. The apparatus according to claim 9 or 10, characterized in that, The determining module is further configured to determine the first decoding algorithm and the priority order in which the first decoding algorithm is executed, based on the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The acquisition module is further configured to acquire a target decoding algorithm set, the target decoding algorithm set including multiple decoding algorithms with priority order, the multiple decoding algorithms including the first decoding algorithm and the second decoding algorithm.

13. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: Based on historical input information and the multiple decoding algorithms, the large model is used to generate reference values ​​for the target indicators corresponding to each of the multiple decoding algorithms. Based on the reference values ​​of the target indicators corresponding to the multiple decoding algorithms, the execution order of each of the multiple decoding algorithms is determined. The target decoding algorithm set is obtained according to the order in which the multiple decoding algorithms are executed.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The large model inference system is deployed in at least one cloud data center managed by the cloud management platform, and each cloud data center is equipped with at least one server.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images