Model reasoning method and device and computing equipment

Abstract

The invention provides a model reasoning method. The method comprises the following steps: receiving input information; generating output information related to the input information by using the large model according to a first decoding algorithm; acquiring a real value of a target index of the large model in the process of generating the output information; under the condition that the difference value between the real value of the target index and the reference value of the target index corresponding to the first decoding algorithm is larger than a preset threshold value, a second decoding algorithm is selected, and the second decoding algorithm is different from the first decoding algorithm; and generating output information related to the input information by using the large model according to a second decoding algorithm. The method can improve the throughput of a large model reasoning system and reduce the reasoning time delay of the large model.

Description

[0001] This application claims priority to the Chinese patent application No. 202410788973.X, filed on June 19, 2024, with the State Intellectual Property Office of China, with the title of “Method, device and computing equipment for model training”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of cloud computing, and more particularly, to a method, device and computing equipment for model inference. BACKGROUND

[0003] A large model refers to a machine learning model with large-scale parameters and complex computing structure, which is usually constructed by a deep neural network and has tens of billions or even hundreds of billions of parameters. A large model inference system refers to a system that uses a trained large model to provide inference services. The system receives text input (e.g., a prompt input by a user or other system) and generates and returns corresponding text output through the decoding process of the large model.

[0004] In a related large model inference system, an autoregressive decoding method is used to provide inference services. Since the autoregressive decoding method can only produce one token output at a time, if a request needs to return multiple outputs, the inference system needs to perform multiple inference processes. This results in a low throughput and long latency of the large model inference system.

[0005] In another related large model inference system, some decoding acceleration algorithms are used to accelerate the inference of the large model. The setting of such decoding acceleration algorithms is one-time, and once a certain decoding acceleration algorithm is selected in the deployment state, the decoding acceleration algorithm will run in the entire running state. Since these decoding acceleration algorithms consume a part of additional computing resources, and the input of the large model inference is usually a large batch, in the case of heavy load and tight computing resources, these decoding acceleration algorithms are likely to cause the performance of the inference system to be worse than the autoregressive decoding algorithm. This results in negative benefits, a low throughput and long latency of the large model inference system.

[0006] Therefore, how to improve the throughput of the large model inference system and reduce the inference latency of the large model becomes a technical problem to be solved urgently. SUMMARY

[0007] The present application provides a method for model inference, which can improve the throughput of the large model inference system and reduce the inference latency of the large model.

[0008] In a first aspect, a model inference method is provided, which is applied to a large model inference system including a large model, and the method comprises: receiving input information as an input of the large model; generating output information related to the input information by using the large model according to a first decoding algorithm; obtaining a real value of a target index of the large model in the process of generating the output information; in a case where a difference between the real value of the target index and a reference value of the target index corresponding to the first decoding algorithm is greater than a preset threshold, selecting a second decoding algorithm, wherein the second decoding algorithm is different from the first decoding algorithm; and generating output information related to the input information by using the large model according to the second decoding algorithm.

[0009] In the above technical solution, a plurality of decoding algorithms can be deployed in an initial deployment stage in the process of model inference, and the large model inference system can dynamically and adaptively adjust the decoding algorithm used by the large model inference system according to the change of the index concerned by the customer, so as to realize stable acceleration of the large model inference, reduce time delay fluctuation, and improve throughput.

[0010] In combination with the first aspect, in some implementations of the first aspect, the priority of the first decoding algorithm is higher than the priority of the second decoding algorithm.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further comprises: determining the first decoding algorithm and the priority order of the first decoding algorithm according to the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further comprises: obtaining the target decoding algorithm set, the target decoding algorithm set including a plurality of decoding algorithms with a priority order, and the plurality of decoding algorithms including the first decoding algorithm and the second decoding algorithm.

[0013] In combination with the first aspect, in some implementations of the first aspect, the reference value of the target index corresponding to each of the plurality of decoding algorithms is generated by using the large model according to historical input information and the plurality of decoding algorithms; the order of execution of each of the plurality of decoding algorithms is determined according to the reference value of the target index corresponding to each of the plurality of decoding algorithms; and the target decoding algorithm set is obtained according to the order of execution of each of the plurality of decoding algorithms.

[0014] In combination with the first aspect, in some implementations of the first aspect, the target index includes any one of the following: compression rate, acceleration ratio, first token delay TTFT, and each output token delay TPOT.

[0015] With reference to the first aspect, in some implementations of the first aspect, the plurality of decoding algorithms includes an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0016] With reference to the first aspect, in some implementations of the first aspect, the large model inference system is deployed in at least one cloud data center managed by a cloud management platform, and each of the cloud data centers is provided with at least one server.

[0017] The second aspect provides a device for model inference, which is applied to a large model inference system including a large model, and the device includes a receiving module, an inference module, an obtaining module, and a determining module. The receiving module is configured to receive input information as an input of the large model. The inference module is configured to generate output information related to the input information by using the large model according to a first decoding algorithm. The obtaining module is configured to obtain an actual value of a target index in a process of generating the output information by the large model. The determining module is configured to select a second decoding algorithm in a case that a difference between the actual value of the target index and a reference value of the target index corresponding to the first decoding algorithm is greater than a preset threshold, wherein the second decoding algorithm is different from the first decoding algorithm. The inference module is further configured to generate output information related to the input information by using the large model according to the second decoding algorithm.

[0018] With reference to the second aspect, in some implementations of the second aspect, a priority of execution of the first decoding algorithm is higher than a priority of execution of the second decoding algorithm.

[0019] With reference to the second aspect, in some implementations of the second aspect, the determining module is further configured to determine the first decoding algorithm and a priority order of execution of the first decoding algorithm according to the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

[0020] With reference to the second aspect, in some implementations of the second aspect, the obtaining module is further configured to obtain a target decoding algorithm set including a plurality of decoding algorithms with a priority order, wherein the plurality of decoding algorithms includes the first decoding algorithm and the second decoding algorithm.

[0021] With reference to the second aspect, in some implementations of the second aspect, the obtaining module is specifically configured to: generate reference values of the target index corresponding to the plurality of decoding algorithms respectively by using the large model according to historical input information and the plurality of decoding algorithms; determine an order of execution of the plurality of decoding algorithms respectively according to the reference values of the target index corresponding to the plurality of decoding algorithms respectively; and obtain the target decoding algorithm set according to the order of execution of the plurality of decoding algorithms respectively.

[0022] In some implementations of the second aspect, in conjunction with the second aspect, the target indicators include any one of: compression rate, speedup ratio, delay of first token TTFT, delay of each output token TPOT.

[0023] In some implementations of the second aspect, in conjunction with the second aspect, the plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0024] In some implementations of the second aspect, in conjunction with the second aspect, the large model inference system is deployed in at least one cloud data center managed by a cloud management platform, and each of the cloud data centers is provided with at least one server.

[0025] It should be understood that the beneficial effects of the second aspect and the various implementations of the second aspect are described in the first aspect and the various implementations of the first aspect, and will not be repeated here.

[0026] In a third aspect, a computing device is provided, including a processor and a memory, and optionally further including an input / output interface. The processor is configured to control the input / output interface to receive and send information, and the memory is configured to store a computer program. The processor is configured to call and run the computer program from the memory, so that the processor executes the method in the first aspect or any possible implementation of the first aspect.

[0027] Optionally, the processor can be a general-purpose processor, which can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which reads software codes stored in the memory to implement the processor. The memory can be integrated in the processor or exist independently outside the processor.

[0028] In a fourth aspect, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, a chip is provided, which obtains instructions and executes the instructions to implement the method in the first aspect or any possible implementation of the first aspect.

[0030] Optionally, as an implementation, the chip includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the method in the first aspect or any possible implementation of the first aspect.

[0031] Optionally, as an implementation form, the chip further comprises a memory, and the memory stores instructions, and the processor is configured to execute the instructions stored in the memory, and the processor is configured to execute the method in the first aspect and any implementation form of the first aspect when the instructions are executed.

[0032] In a sixth aspect, a computer program product is provided, and the computer program product comprises instructions, and the instructions, when executed by a computing device, cause the computing device to perform the method in the first aspect and any implementation form of the first aspect.

[0033] In a seventh aspect, a computer program product is provided, and the computer program product comprises instructions, and the instructions, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect and any implementation form of the first aspect.

[0034] In an eighth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium comprises computer program instructions, and the computer program instructions, when executed by a computing device, cause the computing device to perform the method in the first aspect and any implementation form of the first aspect.

[0035] As examples, the computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0036] Optionally, as an implementation form, the storage medium can be a non-volatile storage medium.

[0037] In a ninth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium comprises computer program instructions, and the computer program instructions, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect and any implementation form of the first aspect.

[0038] As examples, the computer-readable storage includes, but is not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0039] Optionally, as an implementation manner, the storage medium can be a nonvolatile storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic block diagram of a cloud scenario suitable for the embodiments of the present application.

[0041] Figure 2 is a schematic flow chart of a model inference method provided by the embodiments of the present application.

[0042] Figure 3 is a schematic block diagram of a large model inference system provided by the embodiments of the present application.

[0043] Figure 4 is a schematic flow chart of another model inference method provided by the embodiments of the present application.

[0044] Figure 5 is a schematic diagram of a basic setting interface of a large model inference deployment provided by the embodiments of the present application.

[0045] Figure 6 is a schematic diagram of an advanced setting interface of a large model inference deployment provided by the embodiments of the present application.

[0046] Figure 7 is an interface schematic diagram of an ordering of decoding engines corresponding to respective batch sizes recommended by a large model inference system according to the size of an acceleration ratio provided by the embodiments of the present application.

[0047] Figure 8 is a schematic diagram of a large model deployment success interface provided by the embodiments of the present application.

[0048] Figure 9 is a schematic diagram of monitoring and recording the acceleration ratio change of a large model during inference provided by the embodiments of the present application.

[0049] Figure 10 is a schematic block diagram of a model inference apparatus 900 provided by the embodiments of the present application.

[0050] Figure 11 is an architecture schematic diagram of a computing device 1500 provided by the embodiments of the present application.

[0051] Figure 12 is an architecture schematic diagram of a computing device cluster provided by the embodiments of the present application.

[0052] Figure 13 is a schematic diagram of connection between computing devices 1500A and 1500B through a network provided by the embodiments of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the present application will be described below with reference to the drawings.

[0054] The present application will present various aspects, embodiments or features around systems including a plurality of devices, components, modules, etc. It should be understood and appreciated that each system can include additional devices, components, modules, etc., and / or can not include all of the devices, components, modules, etc. discussed in connection with the figures. Furthermore, a combination of these aspects can also be used.

[0055] In addition, in the embodiments of the present application, the words "example" and "for example" are used to mean serving as an example, instance, or illustration. Any implementation described herein as "example" is not necessarily to be construed as preferred or advantageous over other implementations. Rather, use of the word example is intended to present concepts in a concrete manner.

[0056] In the embodiments of the present application, "corresponding" and "relevant" can be used interchangeably at times. It should be noted that when the distinction is not emphasized, the meanings expressed are consistent.

[0057] The business scenarios described in the embodiments of the present application are to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, as the network architecture evolves and new business scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0058] In this specification, the reference to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments unless otherwise indicated. The terms "including," "comprising," "having" and variations thereof mean "including but not limited to," unless expressly specified otherwise.

[0059] In this application, "at least one" means one or more, and "multiple" means two or more. The association relationship between the associated objects is described, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0060] For ease of description, the concepts involved in the embodiments of the present application are explained first.

[0061] 1. Large model

[0062] Large model refers to a machine learning model with large-scale parameters and complex computational structure. These models are usually built by deep neural networks and have tens of billions or even hundreds of billions of parameters. Large models learn complex patterns and features by training massive amounts of data, and have stronger generalization ability, which can make accurate predictions on unseen data.

[0063] In the embodiments of the present application, the large model can include but is not limited to: large language model (LLM), large model using Transformer as the basic model architecture, etc.

[0064] Large language model LLM is a large deep learning model pre-trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, and translation. These models usually contain tens of billions or even more parameters, which are learned from a large amount of text data to predict the next word or generate content related to the given text. Therefore, large language models and their applications in the field of artificial intelligence have become a hot research topic in the global technology field, and the growth of the number of parameters has attracted particular attention, which has jumped from the initial tens of billions to the current hundreds of billions. The increase in the number of parameters enables the model to capture the subtleties of human language more finely and understand the complexity of human language more deeply. Large language models have made significant improvements in absorbing new knowledge, decomposing complex tasks, and aligning text and images. With the continuous maturation of technology, it will continue to expand its application range and provide more intelligent and personalized services to humans, further improving people's lives and production methods.

[0065] 2. Large model inference system

[0066] A large model inference system refers to a system that uses a large model that has been trained to provide inference services. The system receives text input from a user or other system (e.g., a prompt input by the user or other system), processes the input through the decoding process of the large model, and generates and returns corresponding text output.

[0067] Typically, a large model inference system is composed of an input processor, a scheduler, a large model engine, a return processor, and the like.

[0068] The prompt mentioned above generally refers to a piece of text or information that serves as an initial input or guide for the large model to generate tokens. In other words, the prompt can be regarded as the starting point or trigger for the large model to generate tokens. By providing an appropriate prompt, the large model can generate tokens related to the content of the prompt.

[0069] 3. Autoregressive decoding

[0070] Autoregressive decoding is a decoding strategy used by large models to provide inference services. Specifically, autoregressive decoding is token by token, that is, in the process of autoregressive decoding, the probability distribution of the next token is calculated based on the existing context (i.e., the previously generated tokens), and a token is selected as the next output based on this probability distribution. This process is recursive, meaning that each newly generated token becomes a condition for subsequent prediction. For example, in a text generation task, the first token of the sequence is generated first, then the second token is predicted based on the first token, and so on, until the complete sequence is generated.

[0071] It should be understood that the token mentioned above can be a word, character, word, image block, or other type of data unit, depending on the application field and task of the model.

[0072] 4. Large model inference acceleration techniques

[0073] Acceleration techniques for large model inference include, but are not limited to, model compression, decoding methods, underlying optimization, and distributed parallel inference.

[0074] Embodiments of the present application mainly focus on acceleration techniques for decoding methods, mainly using algorithms that can produce multiple token outputs in one decoding step to accelerate inference, thereby achieving the purpose of accelerating the large model inference process.

[0075] Some exemplary commonly used inference acceleration algorithms include, but are not limited to, size model collaborative speculation decoding algorithm, forward-looking decoding algorithm, Medusa decoding algorithm, doll speculation decoding algorithm, etc.

[0076] For example, the size model collaborative speculation decoding algorithm is to generate the output of multiple tokens in advance by a small model, and then verify whether the output of the multiple tokens conforms to the output distribution of the large model by the large model through one calculation, and if it conforms, accept the output of the multiple tokens of the small model as the output of the large model, so as to achieve the purpose of accelerating the inference of the large model.

[0077] For another example, forward-looking decoding is another acceleration method without small model collaboration. The forward-looking decoding drafts possible outputs and verifies possible outputs by adding additional calculation in a decoding step. Similarly, if multiple tokens pass the verification, multiple tokens can be generated in a decoding step, and the purpose of acceleration can also be achieved.

[0078] In recent years, with the rapid development of large models, intelligent applications based on large models have been widely landed and have affected the way people produce and use information, such as ChatGPT, Sora, etc. A representative feature of large models is the huge number of parameters. In order to continuously improve the application effect of large models, current large models are still developing rapidly in the direction of expanding the number of parameters. In 2023, OpenAI announced ChatGPT with 100 billion parameters, and within a few months, GPT4 with 1 trillion parameters was announced. The dialogue effect of the model has improved significantly and has caused continuous uproar in the industry. The huge number of parameters has both achieved large models and hindered the popularization of large models. One of the important reasons is the high-cost service deployment of large models in inference state. Therefore, how to improve the throughput of large model inference and reduce the inference latency of large models is a thorny and important problem.

[0079] In a related large model inference system, a trained large model is generally directly used to provide inference services by using the autoregressive decoding method. The autoregressive decoding method has the characteristic that only one token output can be generated at a time. If a request needs to return 1024 outputs, the inference system needs to perform 1024 inference processes. Each inference process needs to move the parameters of the large model from the storage unit of the xPU to the calculation unit, and this moving process is very slow compared to the calculation process. Therefore, in the large model inference system, the throughput of the large model in the inference process is low, and the latency is long.

[0080] In another related large model inference system, some decoding acceleration algorithms are used to accelerate the inference of large models. For example, by using some additional computing resources, the large model produces multiple token outputs in one decoding step through some decoding acceleration algorithms, thereby reducing the loading and transmission of large model parameters, and achieving the purpose of accelerating the inference process of the large model. The setting of such decoding acceleration algorithms is one-time, and once a certain decoding acceleration algorithm is selected in the deployment state, the decoding acceleration algorithm will run in the entire running state. Since these decoding acceleration algorithms consume a part of additional computing resources, they can only accelerate the inference of large models in a very small batch. The input of large model inference is often a large batch, and in the case of heavy load and tight computing resources, these decoding acceleration algorithms are likely to cause the performance of the inference system to be worse than the autoregressive decoding algorithm, thereby bringing negative benefits.

[0081] In summary, the related large model inference system described above can only set one decoding algorithm at the initial deployment, and the decoding algorithm will run in the entire running state, thereby making the throughput of the large model in the inference process low and the latency long.

[0082] Therefore, the embodiments of the present application provide a model inference method, which can support the deployment of multiple decoding algorithms in the initial deployment stage during the model inference process, so that the decoding algorithm used by the large model inference system can be adaptively adjusted according to the index change, thereby improving the throughput of the large model in the inference process and reducing the inference latency of the large model.

[0083] In a possible implementation manner, the method provided by the embodiments of the present application can be applied to a cloud service scenario, and the method is executed by a cloud management platform in the cloud service scenario. For ease of description, the cloud service scenario is described in detail below. Figure 1

[0084] Figure 1 is a schematic block diagram of a cloud scenario suitable for the embodiments of the present application. As Figure 1 indicated, the cloud scenario can include a cloud management platform 110, an Internet 120, and a client 130.

[0085] As Figure 1 indicated, the cloud management platform 110 is configured to manage the infrastructure providing multiple cloud services. The infrastructure includes multiple cloud data centers, each cloud data center includes multiple servers, and each server includes cloud service resources, which provide corresponding cloud services for tenants.

[0086] ​The cloud management platform 110 can be located in a cloud data center, which can provide an access interface (such as an interface or an application program interface (API)). The tenant can remotely access the access interface to register a cloud account and a password on the cloud management platform 110 and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and the password, the tenant can further select and purchase a virtual machine of a specific specification (processor, memory, disk) on the cloud management platform 110 by paying a fee. After the payment and purchase are successful, the cloud management platform 110 provides a remote login account and password of the purchased virtual machine, and the client 130 can remotely log in to the virtual machine, install and run the application of the tenant in the virtual machine. Therefore, the tenant can create, manage, log in to, and operate the virtual machine in the cloud data center through the cloud management platform 110. The virtual machine can also be referred to as a cloud server (elastic compute service, ECS), an elastic instance (different cloud service providers have different names).

[0087] It should be understood that the tenant of the cloud service can be an individual, an enterprise, a school, a hospital, an administrative organ, and the like.

[0088] The functions of the cloud management platform 110 include but are not limited to a user console, a computing management service, a network management service, a storage management service, an authentication service, and an image management service. The user console provides an interface or an API to interact with the tenant, the computing management service is used to manage servers running virtual machines and containers and bare metal servers, the network management service is used to manage network services (such as gateways and firewalls), the storage management service is used to manage storage services (such as data bucket services), the authentication service is used to manage the account and password of the tenant, and the image management service is used to manage virtual machine images. The tenant uses the client 130 to log in to the cloud management platform 110 through the Internet 120 to manage the rented cloud service.

[0089] Figure 2 is a schematic flowchart of a model inference method provided by an embodiment of the present application. As shown in Figure 2 The method can include steps 210-230, which are described in detail below.

[0090] Step 210: Obtain a target decoding algorithm set corresponding to a target index.

[0091] In the embodiment of the present application, the large model inference system can obtain a target decoding algorithm set corresponding to a target index, and the target decoding algorithm set includes a plurality of decoding algorithms with an execution order.

[0092] The above target indicators can include, but are not limited to, any of the following indicators: compression ratio, speedup ratio, time to first token (TTFT), time per output token (TPOT).

[0093] The meanings of the above indicators are explained in detail as follows.

[0094] 1. Compression ratio

[0095] The current base decoding algorithm is autoregressive decoding, that is, each decoding step can only produce one token output, while the large model decoding acceleration algorithm can produce n (n>=1) tokens output per step by consuming some additional computing resources. The ratio of the two decoding algorithms is the compression ratio (cr), where cr=n / 1. It should be understood that the higher or lower the compression ratio can generally indicate the pros and cons of a decoding acceleration algorithm.

[0096] 2. Speedup ratio

[0097] Suppose the time consumed by each decoding step of autoregressive decoding is t r , and the time consumed by each decoding step of a certain decoding acceleration algorithm is t e , the calculation method of the speedup ratio is When the speedup ratio is equal to 1, it means that the decoding acceleration algorithm cannot bring speedup benefits; when the speedup ratio is less than 1, it means that the running efficiency of the decoding acceleration algorithm is lower than that of the autoregressive decoding algorithm; when the speedup ratio is greater than 1, it means that the decoding acceleration algorithm can achieve the purpose of inference acceleration.

[0098] 3. TTFT

[0099] TTFT refers to the delay of the first token. That is, the delay of the first token from input to output. For example, in online streaming applications, TTFT is the most important indicator, because it determines the user experience.

[0100] 4. TPOT

[0101] TPOT refers to the delay of each output token (excluding the first token). For example, in offline batch applications, TPOT is the most important indicator, because it determines the time of the entire inference process.

[0102] For example, the above multiple decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm. The decoding acceleration algorithm includes, but is not limited to: large model collaborative speculation decoding algorithm, lookahead decoding algorithm, medusa decoding algorithm, Russian nesting doll speculation decoding algorithm, etc.

[0103] In the embodiments of the present application, there are various implementation manners for the large model inference system to obtain the target decoding algorithm set corresponding to the target index, which are not limited in the embodiments of the present application. Two possible implementation manners are introduced as follows.

[0104] Implementation manner 1: The large model inference system obtains a first decoding algorithm set and recommends the first decoding algorithm set to the customer. If the customer does not modify the execution order of each decoding algorithm in the first decoding algorithm set, the customer sends confirmation indication information to the large model inference system. According to the confirmation indication information of the customer, the large model inference system can take the first decoding algorithm set as the target decoding algorithm set.

[0105] Implementation manner 2: The large model inference system obtains a first decoding algorithm set and recommends the first decoding algorithm set to the customer. If the customer modifies the execution order of each decoding algorithm in the first decoding algorithm set according to its own experience, the customer sends a second decoding algorithm set to the large model inference system, which is obtained by modifying the execution order of each decoding algorithm in the first decoding algorithm set. After receiving the second decoding algorithm set sent by the customer, the large model inference system can take the second decoding algorithm set as the target decoding algorithm set.

[0106] The process of the large model inference system obtaining the first decoding algorithm set is described in detail as follows.

[0107] As an example, the large model inference system can perform a trial run on the large model inference system to obtain reference values of target indexes corresponding to a plurality of decoding algorithms respectively. According to the reference values of the target indexes corresponding to the plurality of decoding algorithms respectively, the execution order of each decoding algorithm is determined to obtain a first decoding algorithm set, which includes a plurality of decoding algorithms with execution orders.

[0108] For example, taking the target index as the acceleration ratio, the plurality of decoding algorithms includes decoding algorithm 1, decoding algorithm 2, decoding algorithm 3, and decoding algorithm 4. Assuming that the acceleration ratio corresponding to decoding algorithm 1 is 3.0, the acceleration ratio corresponding to decoding algorithm 2 is 2.0, the acceleration ratio corresponding to decoding algorithm 3 is 2.5, and the acceleration ratio corresponding to decoding algorithm 4 is 1.0. The execution order or priority of the plurality of decoding algorithms from high to low is: decoding algorithm 1, decoding algorithm 3, decoding algorithm 2, and decoding algorithm 4.

[0109] Step 220: According to the value of the target index obtained by the large model in the actual inference process and the reference value of the target index corresponding to each decoding algorithm, the target decoding algorithm is adaptively selected from the plurality of decoding algorithms.

[0110] In the embodiments of the present application, the large model inference system can collect the value of the target index in real time during the actual inference of the large model. And according to the collected value of the target index and the reference value of the target index corresponding to each of the plurality of decoding algorithms, the target decoding algorithm is adaptively selected from the plurality of decoding algorithms.

[0111] In one example, the first decoding algorithm with the highest priority in the target decoding algorithm set can be used as the target decoding algorithm.

[0112] In another example, if the decoding algorithm currently used by the large model inference system is the first decoding algorithm, the large model inference system can also collect the value of the target index in real time. If the switching condition is met, the large model inference system switches the decoding algorithm from the first decoding algorithm to the second decoding algorithm. That is, the second decoding algorithm in the target decoding algorithm set can be used as the target decoding algorithm.

[0113] As an example, the switching condition can be that the first difference is less than a preset threshold, that is, the decoding algorithm can be switched from the first decoding algorithm to the second decoding algorithm when the first difference is less than the preset threshold. Wherein, the first difference is the difference between the reference value of the target index corresponding to the first decoding algorithm and the value of the target index obtained by the large model during the inference process.

[0114] Step 230: obtaining the output information of the large model based on the input information of the large model and the target decoding algorithm.

[0115] In the embodiments of the present application, the large model inference system can receive input information, and obtain the output information of the large model based on the input information of the large model and the target decoding algorithm.

[0116] In the above technical solution, the plurality of decoding algorithms can be deployed in advance in the initial deployment stage during the model inference process. The large model inference system can dynamically and adaptively adjust the decoding algorithm used by the large model inference system according to the changes of the index concerned by the customer, so as to realize the stable acceleration of the large model inference, reduce the delay fluctuation, and improve the throughput.

[0117] Figure 3 is a schematic block diagram of a large model inference system provided by the embodiments of the present application. As an example, as shown in Figure 3 The system can include an input receiving unit, a scheduler, a large model engine, a decoding engine monitoring unit, a decoding engine switching unit, a plurality of decoding engines (for example, decoding engine 1, decoding engine 2, decoding engine 3), a draft pool, an output returning unit, etc.

[0118] The input receiving unit is configured to receive the input request in the running state and transmit the received request to the dispatcher. The dispatcher transmits the request to the large model engine according to the current resource usage. The large model engine can perform inference according to the currently set decoding engine 1 (which executes the currently set decoding algorithm 1).

[0119] During the inference process, the decoding engine monitoring unit obtains the changes of various indicators and determines whether the decoding algorithm needs to be adjusted, i.e., whether the currently set decoding engine 1 needs to be switched. For example, taking the speedup ratio as an indicator, if the current speedup ratio is greater than or equal to the predetermined speedup ratio of the decoding engine 1, the current decoding engine 1 is still used for inference. If the current speedup ratio is less than the predetermined speedup ratio of the decoding engine 1, the decoding engine switching unit switches to the decoding engine 2 that consumes less resources, and the native autoregressive decoding engine serves as the final fallback solution.

[0120] After the inference process, the inference system can return the final output to the user through the output returning unit.

[0121] The specific implementation process of the model inference method provided by the embodiments of the present application will be described in detail below. Figure 4 It should be understood that, Figure 4 the examples are only intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to Figure 4 the specific values or specific scenarios shown in the examples. Those skilled in the art can obviously make various equivalent modifications or changes to the examples given below, and such modifications and changes also fall within the scope of the embodiments of the present application. Figure 4

[0122] Figure 4 is a schematic flowchart of another model inference method provided by the embodiments of the present application. As Figure 4 shown, the method can include steps 410-440, which will be described in detail below.

[0123] Step 410: The large model inference system obtains the large model parameters used by the customer when deploying the large model inference service.

[0124] In the embodiments of the present application, the customer can submit the large model that needs to be run or used when deploying the large model inference service, for example, submit the parameters of the large model that needs to be run or used to the large model inference system.

[0125] ​As an example, the parameters of the large model submitted by the customer to the large model inference system can come from a local place, or can also come from an online storage system such as an object storage service (OBS).

[0126] Optionally, the customer can also set some other basic parameters of the large model, such as: the maximum inference length supported by the large model, whether the large model uses fp16 precision, etc.

[0127] For example, Figure 5 A schematic diagram of a basic setting interface of a large model inference deployment provided by the large model inference system. As shown in the figure, Figure 4 the customer can upload the parameters of the large model and some other basic parameters of the large model through the interface.

[0128] For example, if the parameters of the large model come from an online storage system, the customer can input "obs: / / url" in the first dialog box of the interface as shown in Figure 5 and click the "Upload Model Parameters" button. The large model inference system can obtain the parameters of the large model from the OBS based on the uniform resource locator (URL).

[0129] For another example, the customer can also set the maximum inference length supported by the large model to "4098" and whether the large model uses fp16 precision to "Yes" in the interface as shown in Figure 5 .

[0130] Step 420: The large model inference system obtains other advanced settings of the customer for the large model inference service.

[0131] In the embodiments of the present application, the customer can also make some other advanced settings when deploying the large model inference service, for example, the customer sets some decoding methods used by the large model when performing inference, which includes a basic decoding method and a plurality of decoding acceleration methods.

[0132] As an example, the decoding acceleration method can include but is not limited to: a large model speculation decoding algorithm, a look-ahead decoding algorithm, a Medusa decoding algorithm, a doll speculation decoding algorithm, etc.

[0133] For example, Figure 6 A schematic diagram of an advanced setting interface of a large model inference deployment provided by the large model inference system. As shown in the figure, Figure 6 the customer can set a plurality of decoding algorithms used by the large model when performing inference through the interface.

[0134] For example, as shown in Figure 6As shown, the customer sets the base decoding algorithm used by the large model when performing inference to the autoregressive decoding algorithm. Exemplarily, as shown in Figure 6 As shown, the autoregressive decoding engine is configured to perform the autoregressive decoding algorithm.

[0135] For another example, the customer sets the decoding acceleration algorithm used by the large model when performing inference to the large model speculation decoding algorithm, the look-ahead decoding algorithm, and the medusa decoding algorithm. Exemplarily, as shown in Figure 6 As shown, the large model speculation decoding engine is configured to perform the large model speculation decoding algorithm, the look-ahead decoding engine is configured to perform the look-ahead decoding algorithm, and the medusa decoding engine is configured to perform the medusa decoding algorithm.

[0136] It should be noted that according to different decoding acceleration methods, the customer also needs to upload or provide related support files.

[0137] For example, for the large model speculation decoding algorithm, the customer also needs to upload the parameters of the small model used in the large model speculation decoding algorithm through the interface as shown in Figure 6 Exemplarily, assuming that the parameters of the small model come from an online storage system, the customer can input “obs: / / url” in the interface as shown in Figure 5 and click the “Upload Small Model” button, and the large model inference system can obtain the parameters of the small model used in the large model speculation decoding algorithm from the OBS based on the URL.

[0138] For another example, for the medusa decoding algorithm, the customer also needs to upload some additional parameters used in the medusa decoding algorithm through the interface as shown in Figure 6 Exemplarily, assuming that the additional parameters come from an online storage system, the customer can input “obs: / / url” in the interface as shown in Figure 5 and click the “Upload Additional Model” button, and the large model inference system can obtain the additional parameters used in the medusa decoding algorithm from the OBS based on the URL.

[0139] It should be understood that for the look-ahead decoding algorithm, since the large model inference system does not need related support files when performing the algorithm, the customer also does not need to upload the files related to the look-ahead decoding algorithm in the interface as shown in Figure 6 .

[0140] Step 430: The large model inference system obtains the ranking of the various decoding engines corresponding to each batch size under the target indicators of the final deployment.

[0141] In the embodiments of the present application, the large model inference system can run for a period of time after the customer completes the above configuration, and give the indicators of different decoding engines under the setting of each batch size. And show the customer the optimal ranking of the decoding engines corresponding to each batch size under different indicators, for the customer's reference.

[0142] As an example, the above indicators can include but are not limited to: compression ratio, speedup ratio, time to first token (TTFT), time per output token (TPOT).

[0143] In a possible implementation manner, the customer can select the target indicator of interest from the indicators, and manually intervene or adjust the optimal ranking of the decoding engines corresponding to each batch size recommended by the large model inference system under the target indicator of interest, to form a final deployment scheme.

[0144] For example, if the customer does not adjust the optimal ranking of the decoding engines corresponding to each batch size recommended by the large model inference system, the optimal ranking of the decoding engines corresponding to each batch size recommended by the large model inference system can be used as the final deployment scheme.

[0145] For another example, if the customer adjusts the optimal ranking of the decoding engines corresponding to each batch size recommended by the large model inference system, and obtains the final ranking of the decoding engines corresponding to each batch size, the final ranking of the decoding engines corresponding to each batch size can be used as the final deployment scheme.

[0146] For example, taking the speedup ratio as the target indicator, Figure 7 the large model inference system recommends the ranking of the decoding engines corresponding to each batch size according to the size of the speedup ratio.

[0147] For example, Figure 7As shown, for a batch size of 1-4 prompts, the speedup of accelerated decoding engine 3 is 3.0, accelerated decoding engine 2 is 2.5, accelerated decoding engine 1 is 2.0, and the autoregressive decoding engine is 1.0. Therefore, based on the speedup of each decoding engine, for a batch size of 1-4 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 3, accelerated decoding engine 2, accelerated decoding engine 1, and autoregressive decoding engine.

[0148] For example, such as Figure 7 As shown, for a batch size of 5-12 prompts, the speedup of accelerated decoding engine 3 is 2.5, accelerated decoding engine 2 is 2.2, accelerated decoding engine 1 is 1.8, and the autoregressive decoding engine is 1.0. Therefore, based on the speedup of each decoding engine, for this batch size of 5-12 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 3, accelerated decoding engine 2, accelerated decoding engine 1, and autoregressive decoding engine.

[0149] For example, such as Figure 7 As shown, for a batch size of 13-32 prompts, the speedup ratio for accelerated decoding engine 2 is 1.8, for accelerated decoding engine 3 it is 1.5, for accelerated decoding engine 1 it is 1.3, and for the autoregressive decoding engine it is 1.0. Therefore, based on the speedup ratios of each decoding engine, for this batch size of 13-32 prompts, the recommended decoding engines for the large model inference system are, in order: accelerated decoding engine 2, accelerated decoding engine 3, accelerated decoding engine 1, and autoregressive decoding engine.

[0150] For example, such as Figure 7As shown, for the input of the batch size greater than 33 prompts, the value of the speedup ratio corresponding to the autoregressive decoding engine is 1.0, the value of the speedup ratio corresponding to the accelerated decoding engine 1 is 0.9, the value of the speedup ratio corresponding to the accelerated decoding engine 2 is 0.8, and the value of the speedup ratio corresponding to the accelerated decoding engine 3 is 0.8. Therefore, according to the speedup ratios of the decoding engines, the decoding engine recommended by the large model inference system in the batch size greater than 33 prompts is in turn: the autoregressive decoding engine, the accelerated decoding engine 1, the accelerated decoding engine 2, and the accelerated decoding engine 3.

[0151] It should be noted that the autoregressive decoding engine performs an autoregressive decoding algorithm, and the accelerated decoding engine 1, the accelerated decoding engine 2, and the accelerated decoding engine 3 perform different decoding acceleration algorithms. For specific descriptions of the decoding acceleration algorithms, please refer to the foregoing descriptions, which will not be repeated here.

[0152] It should be understood that, as Figure 7 shown, the interface further includes a manual setting, and the customer can manually intervene or adjust the optimal order of the plurality of decoding engines corresponding to each batch size recommended by the large model inference system to form a final deployment scheme through the manual setting.

[0153] In the embodiment of the present application, taking the speedup ratio index as an example, the large model inference system can also adaptively switch the current decoding engine to a more suitable decoding engine according to the switching threshold value according to the change of the speedup ratio corresponding to each decoding engine in the running state.

[0154] For example, as Figure 7 shown, the user can use the switching threshold value set automatically by the system, or can also manually set the switching threshold value, which is not specifically limited in the embodiment of the present application.

[0155] In the embodiment of the present application, after the above operation is completed, it can be considered that the large model is successfully deployed, and the large model inference system can provide the customer with an interface as Figure 8 shown. As an example, Figure 8 shown, the interface can include an indication (for example, 100%) that the model deployment is successful, and provide an API (for example, https:service-api) for the customer to call.

[0156] Step 440: The large model inference system adaptively switches the current decoding engine to a more suitable decoding engine according to the switching threshold value according to the change of the target index of the large model in inference.

[0157] In the embodiments of the present application, when the large model is in the running state of inference, the large model inference system can also determine the corresponding batch size of the target index according to the number of prompts of the inference input, and sort the decoding engine with the highest priority among the decoding engines corresponding to the batch size.

[0158] The large model inference system described above can also monitor the changes of the target index of the large model in real time when inferring, and adaptively switch the current decoding engine to a more suitable decoding engine according to the switching threshold value according to the changes of the target index. For example, taking the target index as the acceleration ratio, as shown in Figure 9 The acceleration ratio monitoring unit in the large model inference system can monitor and record the changes of the target index of the large model in real time when inferring.

[0159] For example, as shown in Figure 9 When the acceleration ratio is stable at a high level, the large model inference system can maintain the currently used decoding engine. For another example, when it is detected that the computing resources are tight, the load is heavy, and the acceleration ratio is in a state of obvious decline, it is necessary to switch to a decoding engine with less resources and lower acceleration ratio, and until the autoregressive decoding engine is switched, so that the stable acceleration of the large model inference system can be ensured.

[0160] For example, taking the target index as the acceleration ratio index, and the batch size interval corresponding to the input prompt as 5-12. The decoding engines deployed by the customer are as follows: acceleration decoding engine 3, acceleration decoding engine 2, acceleration decoding engine 1, and autoregressive decoding engine. For example, when the large model is in the running state of inference, the large model inference system preferentially executes the corresponding decoding acceleration algorithm through the acceleration decoding engine 3 to generate the corresponding output information. For another example, if the large model inference system detects that the current acceleration ratio is in a state of obvious decline during the process of providing inference service by the large model, it is necessary to switch the currently used acceleration decoding engine 3 to the acceleration decoding engine 2, and the acceleration decoding engine 2 executes the corresponding decoding acceleration algorithm to generate the corresponding output information. For another example, if the large model inference system detects that the current acceleration ratio is still in a state of obvious decline during the process of providing inference service by the large model, it is necessary to switch the currently used acceleration decoding engine 2 to the acceleration decoding engine 1, and so on, until the autoregressive decoding engine is switched.

[0161] In the embodiments of the present application, the large model inference system can have system maintenance personnel decide whether to switch during the switching process, or the large model inference system can also switch according to the set switching threshold value.

[0162] As an example, during the switching process, the large model inference system first sends a notification to the system maintenance personnel. If the system maintenance personnel make a decision within a given time, their action takes precedence. Otherwise, the large model inference system automatically switches the decoding engine based on the set switching threshold.

[0163] In the above technical solution, the decoding engine monitoring unit can monitor changes in various indicators such as speedup ratio when the system is running at full resource capacity, prevent the deterioration of the inference system's acceleration effect, stabilize the inference acceleration effect to the greatest extent, and ensure high throughput and low latency.

[0164] The above text combined Figures 1 to 9 The method provided in the embodiments of this application is described in detail below. Figures 10-13 The embodiments of the apparatus of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0165] Figure 10 This is a schematic block diagram of a model reasoning apparatus 1000 provided in an embodiment of this application. The apparatus 1000 can be implemented by software, hardware, or a combination of both. The apparatus 1000 provided in this embodiment of the application can implement the embodiments of this application. Figure 2 or Figure 4 The method flow shown includes the following: the device 1000 comprises a receiving module 1010, an inference module 1020, an acquisition module 1030, and a determination module 1040. The receiving module 1010 receives input information, which serves as the input to the large model. The inference module 1020 generates output information related to the input information using the large model according to a first decoding algorithm. The acquisition module 1030 acquires the actual value of the target indicator during the generation of the output information by the large model. The determination module 1040 selects a second decoding algorithm, which is different from the first decoding algorithm, if the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold. The inference module also generates output information related to the input information using the large model according to the second decoding algorithm.

[0166] Optionally, the first decoding algorithm is executed with a higher priority than the second decoding algorithm.

[0167] Optionally, the determining module 1040 is further configured to determine the first decoding algorithm and the priority order in which the first decoding algorithm is executed, based on the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

[0168] Optionally, the obtaining module 1030 is further configured to obtain the target decoding algorithm set, the target decoding algorithm set comprising a plurality of decoding algorithms in a priority order, the plurality of decoding algorithms comprising the first decoding algorithm and the second decoding algorithm.

[0169] Optionally, the obtaining module 1030 is specifically configured to: generate, according to historical input information and the plurality of decoding algorithms, reference values of the target indicators corresponding to the plurality of decoding algorithms respectively by using the large model; determine an order in which the plurality of decoding algorithms are executed according to the reference values of the target indicators corresponding to the plurality of decoding algorithms respectively; and obtain the target decoding algorithm set according to the order in which the plurality of decoding algorithms are executed.

[0170] Optionally, the target indicators comprise any one of: a compression rate, an acceleration ratio, a first token delay TTFT, and a per-output token delay TPOT.

[0171] Optionally, the plurality of decoding algorithms comprise an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

[0172] Optionally, the large model inference system is deployed in at least one cloud data center managed by a cloud management platform, and each cloud data center is provided with at least one server.

[0173] The apparatus 1000 herein can be embodied in the form of functional modules. The term “module” herein can be implemented in the form of software and / or hardware, and is not limited in this regard.

[0174] For example, the “module” can be a software program, a hardware circuit, or a combination of both, which implements the above functions. For example, the receiving module 1010 is taken as an example, and the implementation of the receiving module 1010 is introduced. Similarly, the implementation of other modules, such as the inference module 1020, the obtaining module 1030, the determining module 1040, and the recommendation module, can refer to the implementation of the receiving module 1010.

[0175] As an example of a software functional unit, the receiving module 1010 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the receiving module 1010 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0176] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between VPCs.

[0177] As an example of a hardware functional unit, the receiving module 1010 can include at least one computing device, such as a server, etc. Alternatively, the receiving module 1010 can also be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented as a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0178] The multiple computing devices included in the receiving module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 1010 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 1010 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0179] Therefore, the modules of the various examples described in the embodiments of the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0180] It should be noted that: the apparatus provided in the above embodiments is used to execute the above method, and the above-mentioned functions are only exemplified by the division of the above-mentioned functional modules. In actual application, the above-mentioned functions can be completed by different functional modules according to the needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above-described functions. For example, the receiving module 1010 can be used to execute any step in the above method, the reasoning module 1020 can be used to execute any step in the above method, the acquisition module 1030 can be used to execute any step in the above method, the determination module 1040 can be used to execute any step in the above method, and the recommendation module can be used to execute any step in the above method. The steps responsible for implementation by the receiving module 1010, the reasoning module 1020, the acquisition module 1030, the determination module 1040, and the recommendation module can be specified as needed, and the above-mentioned apparatus can be implemented by the receiving module 1010, the reasoning module 1020, the acquisition module 1030, the determination module 1040, and the recommendation module to implement different steps in the above method to implement all the functions of the above-mentioned apparatus.

[0181] In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments above, which will not be repeated here.

[0182] The method provided by the embodiments of the present application can be executed by a computing device, which can also be referred to as a computer system. The computer system includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory, and a memory control unit, and the functions and structures of the hardware are described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer includes application programs such as a browser, an address book, word processing software, and instant messaging software. Optionally, the computer system is a handheld device such as a smartphone or a terminal device such as a personal computer, and the present application is not particularly limited as long as the method provided by the embodiments of the present application can be executed. The execution subject of the method provided by the embodiments of the present application can be a computing device, or a functional module in the computing device that can call and execute a program.

[0183] The embodiments of the present application are described in detail below. Figure 11 A computing device provided by the embodiments of the present application is described in detail.

[0184] Figure 11 FIG. 1 is a schematic diagram of an architecture of a computing device 1500 provided by the embodiments of the present application. The computing device 1500 can be a server or a computer or other device with computing capability. Figure 11 The computing device 1500 shown includes at least one processor 1510 and a memory 1520.

[0185] It should be understood that the number of processors and memories in the computing device 1500 is not limited by the present application.

[0186] The processor 1510 executes instructions in the memory 1520, so that the computing device 1500 implements the method provided by the present application. Alternatively, the processor 1510 executes instructions in the memory 1520, so that the computing device 1500 implements the functional modules provided by the present application, thereby implementing the method provided by the present application.

[0187] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiving module such as but not limited to a network interface card and a transceiver to implement communication between the computing device 1500 and other devices or communication networks.

[0188] Optionally, the computing device 1500 also includes a system bus 1540, wherein the processor 1510, the memory 1520 and the communication interface 1530 are connected with the system bus 1540 respectively. The processor 1510 can access the memory 1520 through the system bus 1540, for example, the processor 1510 can read and write data in the memory 1520 or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0189] In one possible implementation, the function of the processor 1510 is mainly to interpret the instructions (or code) of the computer program and process the data in the computer software. The instructions of the computer program and the data in the computer software can be saved in the memory 1520 or the cache 1516.

[0190] Optionally, the processor 1510 can be an integrated circuit chip with a processing capability of signals. As an example but not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0191] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514.

[0192] Optionally, processing units 1512, also referred to as cores or core complexes, are the most important components of a processor. Processing units 1512 are manufactured from single-crystal silicon using certain production processes. All calculations, command receiving, command storage, and data processing of a processor are performed by cores. Processing units independently run program instructions and use the ability of parallel computing to speed up program running. Various processing units have fixed logical structures, for example, a processing unit includes logical units such as a level one cache, a level two cache, an execution unit, an instruction stage unit, and a bus interface.

[0193] In one implementation, memory control units 1514 are configured to control data interaction between memory 1520 and processing units 1512. Specifically, memory control units 1514 receive memory access requests from processing units 1512 and control access to memory based on the memory access requests. By way of example and not limitation, a memory control unit is a memory management unit (MMU) or the like.

[0194] In one implementation, each memory control unit 1514 addresses memory 1520 through a system bus. An arbiter (not shown in FIG. 15) is configured in the system bus and is responsible for handling and coordinating competing access of multiple processing units 1512. Figure 11

[0195] In one implementation, processing units 1512 and memory control units 1514 are communicatively connected through internal connections, such as address lines, of a chip, so as to realize communication between processing units 1512 and memory control units 1514.

[0196] Optionally, each processor 1510 further includes a cache 1516, which is a buffer for data exchange (referred to as a cache). When a processing unit 1512 needs to read data, the processing unit 1512 first searches for the required data in the cache. If the data is found, the processing unit 1512 directly executes the data. If the data is not found, the processing unit 1512 searches for the data in memory. Because the cache runs much faster than memory, the cache helps processing units 1512 run faster.

[0197] Memory 1520 can provide a running space for a process in computing device 1500. For example, a computer program (specifically, a code of the program) for generating a process is stored in memory 1520. After the computer program is run by a processor to generate a process, the processor allocates a corresponding storage space for the process in memory 1520. Further, the storage space further includes a text segment, an initialized data segment, a bit initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during running of the process, such as intermediate data, or process data, and the like, in the storage space corresponding to the process.​

[0198] Optionally, memory, also called RAM, is used to temporarily store data for processing by processor 1510. Processor 1510 moves data to and from memory for processing as long as the computer is in operation.

[0199] By way of example, and not limitation, memory 1520 is volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory can be read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which acts as external cache. By way of example and not limitation, many forms of RAM exist, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). The system and method described herein can be stored on or transmitted across one or more of these and any other suitable types of memory.

[0200] The structure of the computing device 1500 listed above is only an example, and the application is not limited thereto. The computing device 1500 of the embodiments of the application includes various hardware in the prior art computer system, for example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk memories and the like. Those skilled in the art should understand that the computing device 1500 can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the above computing device 1500 can also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above computing device 1500 can also only include devices necessary for the embodiments of the application, and does not necessarily include all the devices shown in the prior art. Figure 11

[0201] The embodiments of the application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0202] As shown in Figure 12 , the computing device cluster includes at least one computing device 1500. The same instructions for executing the above method can be stored in the memory 1520 of one or more computing devices 1500 in the computing device cluster.

[0203] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster can also respectively store partial instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can collectively execute the instructions of the above method.

[0204] It should be noted that the memories 1520 of different computing devices 1500 in the computing device cluster can store different instructions, respectively for executing part of the functions of the above apparatus. That is, the instructions stored in the memories 1520 of different computing devices 1500 can implement the functions of one or more modules in the above apparatus.

[0205] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. Wherein, the network can be a wide area network or a local area network, etc. Figure 13 A possible implementation is shown. As shown in Figure 13 , two computing devices 1500A and 1500B are connected through a network. Specifically, the communication interface in each computing device is connected to the network.

[0206] It should be understood that Figure 13 ​The functions of the computing device 1500A shown in FIG. 15A can also be completed by a plurality of computing devices 1500. Similarly, the functions of the computing device 1500B shown in FIG. 15B can also be completed by a plurality of computing devices 1500.

[0207] In this embodiment, a computer program product containing instructions is also provided. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When it is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to realize the functions of the apparatus provided above.

[0208] In this embodiment, a computer readable storage medium is also provided. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which, when executed on a computing device, cause the computing device to perform the method provided above.

[0209] It should be understood that the size of the sequence number of each process described above in various embodiments of the present application does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0210] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, apparatus and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0212] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0213] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0214] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0215] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), etc.

[0216] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for model reasoning, characterized in that, The method is applied to a large-model inference system, wherein the large-model inference system includes a large model, and the method includes: Receive input information, which serves as the input to the large model; According to the first decoding algorithm, output information related to the input information is generated using the large model; Obtain the actual values ​​of the target indicators during the process of generating the output information by the large model; If the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, a second decoding algorithm is selected, wherein the second decoding algorithm is different from the first decoding algorithm. According to the second decoding algorithm, output information related to the input information is generated using the large model.

2. The method according to claim 1, characterized in that, The first decoding algorithm is executed with a higher priority than the second decoding algorithm.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the reference values ​​of the target indicators corresponding to the first decoding algorithm and the target indicators corresponding to the second decoding algorithm, the first decoding algorithm and the priority order in which the first decoding algorithm is executed are determined.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a target decoding algorithm set, the target decoding algorithm set including multiple decoding algorithms with priority order, the multiple decoding algorithms including the first decoding algorithm and the second decoding algorithm.

5. The method according to claim 4, characterized in that, The set of target decoding algorithms includes: Based on historical input information and the multiple decoding algorithms, the large model is used to generate reference values ​​for the target indicators corresponding to each of the multiple decoding algorithms. Based on the reference values ​​of the target indicators corresponding to the multiple decoding algorithms, the execution order of each of the multiple decoding algorithms is determined. The target decoding algorithm set is obtained according to the order in which the multiple decoding algorithms are executed.

6. The method according to any one of claims 1 to 5, characterized in that, The target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

7. The method according to any one of claims 4 to 6, characterized in that, The plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

8. The method according to any one of claims 1 to 7, characterized in that, The large model inference system is deployed in at least one cloud data center managed by the cloud management platform, and each cloud data center is equipped with at least one server.

9. A device for model reasoning, characterized in that, The device is applied to a large-model inference system, the large-model inference system including a large model, and the device includes: A receiving module is used to receive input information, which serves as the input to the large model; The inference module is used to generate output information related to the input information using the large model based on the first decoding algorithm; The acquisition module is used to acquire the actual values ​​of the target indicators during the process of generating the output information by the large model; The determining module is used to select a second decoding algorithm when the difference between the actual value of the target indicator and the reference value of the target indicator corresponding to the first decoding algorithm is greater than a preset threshold, wherein the second decoding algorithm is different from the first decoding algorithm; The reasoning module is further configured to generate output information related to the input information using the large model based on the second decoding algorithm.

10. The apparatus according to claim 9, characterized in that, The first decoding algorithm is executed with a higher priority than the second decoding algorithm.

11. The apparatus according to claim 9 or 10, characterized in that, The determining module is further configured to determine the first decoding algorithm and the priority order in which the first decoding algorithm is executed, based on the reference value of the target index corresponding to the first decoding algorithm and the reference value of the target index corresponding to the second decoding algorithm.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The acquisition module is further configured to acquire a target decoding algorithm set, the target decoding algorithm set including multiple decoding algorithms with priority order, the multiple decoding algorithms including the first decoding algorithm and the second decoding algorithm.

13. The apparatus according to claim 12, characterized in that, The acquisition module is specifically used for: Based on historical input information and the multiple decoding algorithms, the large model is used to generate reference values ​​for the target indicators corresponding to each of the multiple decoding algorithms. Based on the reference values ​​of the target indicators corresponding to the multiple decoding algorithms, the execution order of each of the multiple decoding algorithms is determined. The target decoding algorithm set is obtained according to the order in which the multiple decoding algorithms are executed.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The target metrics include any of the following: compression ratio, speedup ratio, latency TTFT of the first token, and latency TPOT of each output token.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The plurality of decoding algorithms include an autoregressive decoding algorithm and at least one decoding acceleration algorithm.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The large model inference system is deployed in at least one cloud data center managed by the cloud management platform, and each cloud data center is equipped with at least one server.

17. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 8.

19. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 8.

Images