Model inference method and system, and computing device

Abstract

A model inference method and system. The method comprises: receiving text prompt information, the text prompt information being used for generating text information; determining at least one draft verification model from a draft verification model pool; determining at least one corresponding draft generation model from a draft generation model pool; acquiring from a draft pool a target draft related to the text prompt information, a plurality of drafts being stored in the draft pool and comprising a draft generated by the at least one draft generation model; verifying the target draft by means of the at least one draft verification model to obtain a verification result; and, on the basis of the verification result, generating text information and feeding back same. The method can improve model inference efficiency.

Description

Model reasoning method, system and computing device

[0001] This application claims priority to the Chinese patent application with application number 202410263183.X filed with the State Intellectual Property Office of China on March 7, 2024, and priority to the Chinese patent application with invention name “Method, device and computing device for accelerating model reasoning”, as well as priority to the Chinese patent application with application number 202410381622.7 filed with the State Intellectual Property Office of China on March 29, 2024, and priority to the Chinese patent application with invention name “Method, system and computing device for model reasoning”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and more specifically, to a method, system, and computing device for model reasoning. Background Art

[0003] In recent years, with the rapid development of large models, intelligent applications built on them have been widely implemented, influencing the way people generate and use information. A hallmark of large models is their massive number of parameters. To continuously improve their effectiveness, they are currently undergoing rapid development, focusing on increasing their parameter count. While this massive number of parameters has made large models a success, it has also become a significant constraint to their widespread adoption. A key factor is the high cost of deploying large models as a service for inference. Due to the large number of parameters, large models require extensive computing resources during inference. However, due to the limitations of the token-by-token generation method of autoregressive decoding, computing resource utilization during inference is low. Ensuring that models fully utilize computing resources during inference, improving model inference throughput, and reducing the cost of large model inference is a challenging and important issue.

[0004] In the scheme of reasoning based on relevant generative models, in the two key stages of Draft generation and Draft verification, the Draft generation model and the Draft verification model have a strong coupling relationship in terms of collaboration and correspondence. This strong coupling relationship is solidified when the program starts, resulting in poor flexibility in the collaboration and correspondence between the Draft generation model and the Draft verification model, which reduces the efficiency of model reasoning.

[0005] Therefore, how to improve the efficiency of model reasoning has become a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The present application provides a method, system and computing device for model reasoning, which can improve the efficiency of model reasoning.

[0007] In a first aspect, a method for model reasoning is provided, the method comprising: determining at least one Draft verification model from a Draft verification model pool, wherein the Draft verification model pool includes multiple Draft verification models; determining at least one Draft generation model corresponding to the at least one Draft verification model from a Draft generation model pool, wherein the Draft generation model pool includes multiple Draft generation models; receiving text prompt information input by a first user, wherein the text prompt information is used to generate text information; obtaining a target Draft related to the text prompt information from a Draft pool, wherein the Draft pool stores multiple Drafts, wherein the multiple Drafts include a Draft generated by the at least one Draft generation model; verifying the target Draft using the at least one Draft verification model and obtaining a verification result. The text information is generated based on the verification result and fed back to the first user.

[0008] As an example, the above text prompt information may be a prompt for input, which generally refers to a piece of text or information, serving as an initial input or guide for the generation model, and used to guide the generation model to generate Draft.

[0009] The at least one Draft verification model determined above may be a manufacturer's built-in Draft verification model in the Draft verification model pool, or it may be a Draft verification model obtained by the user after fine-tuning the manufacturer's built-in Draft verification model in the Draft verification model pool based on the user's own data, or it may be the user's own Draft verification model (the user docks his own Draft verification model to the Draft verification model pool through the docking interface).

[0010] The at least one Draft generation model determined above may be a manufacturer's built-in Draft generation model in the Draft generation model pool, or it may be a Draft generation model obtained by the user after fine-tuning the manufacturer's built-in Draft generation model in the Draft generation model pool based on the user's own data, or it may be the user's own Draft generation model (the user docks his own Draft generation model to the Draft generation model pool through the docking interface).

[0011] In the above technical solution, the Draft generation model and the Draft verification model can be completely decoupled in terms of correspondence and collaboration. The Draft generation model and the Draft verification model can be dynamically combined and matched in any proportion. The correspondence and collaboration between the Draft generation model and the Draft verification model are not fixed and are more flexible, thereby improving the efficiency of model reasoning.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the correspondence modes between the at least one Draft verification model and the at least one Draft generation model include: one-to-one, one-to-many, many-to-one, and many-to-many.

[0013] In the above technical solution, there are multiple corresponding relationships between at least one Draft verification model and the at least one Draft generation model, such as one-to-one, one-to-many, many-to-one, many-to-many, etc., and they can be switched in real time and dynamically combined.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the at least one Draft generation model includes a Draft synchronous generation model, and the method further includes: obtaining Draft generation request information based on the text prompt information; sending the Draft generation request information to the Draft synchronous generation model, the Draft synchronous generation model is used to generate a Draft based on the Draft generation request information; and storing the Draft generated by the Draft synchronous generation model in the Draft pool.

[0015] In the above technical solution, the Draft pool can record and manage the Drafts generated by the Draft generation model during the operation of the system, so that the Drafts recorded in the Draft pool can be used for subsequent reasoning, allowing the Draft to be reused, avoiding the generated Draft being only used for the reasoning verification of the current request. To a certain extent, it can reduce the computing power and time occupied by Draft generation, thereby further improving the efficiency of model reasoning and reducing the cost of model reasoning.

[0016] In combination with the first aspect, in some implementations of the first aspect, the Draft pool further stores attribute information of the Draft.

[0017] In the above technical solution, the attribute information of the Draft can also be recorded in the Draft pool, which provides an effective Draft screening mechanism for subsequent reasoning, thereby improving the efficiency of model reasoning and reducing the cost of model reasoning.

[0018] In combination with the first aspect, in certain implementations of the first aspect, a target Draft related to the text prompt information is obtained from the Draft pool based on the attribute information of the Draft stored in the Draft pool.

[0019] In the above technical solution, more effective Drafts can be screened and retrieved for different text prompt information (for example, prompt) based on the attribute information of the Draft stored in the Draft pool.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: retrieving and / or updating the Draft stored in the Draft pool based on the attribute information of the Draft stored in the Draft pool.

[0021] In the above technical solution, the Draft pool can be self-maintained and automatically updated based on the attribute information of the Draft stored in the Draft pool.

[0022] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: operating the Draft generation model included in the Draft generation model pool according to the attribute information of the Draft stored in the Draft pool, and the operation includes: activation, sleep, and release.

[0023] In the above technical solution, based on the attribute information of the Draft stored in the Draft pool, the Draft generation model included in the Draft generation model pool can be activated, dormant, released, etc., thereby improving the efficiency of model reasoning.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the attribute information of the Draft includes at least one of the following: the scope of the Draft, the content, the label of the corresponding Draft generation model, the number of times it is called, the number of times it is hit, and a list of text prompt information of the hit.

[0025] In the above technical solution, the multi-dimensional attribute information of Draft can be used to make Draft recall more accurate and bring a higher hit rate in the Draft verification stage.

[0026] In combination with the first aspect, in certain implementations of the first aspect, the Draft generation request information also includes the number of Drafts generated, and the number of Drafts generated is determined based on the usage of model reasoning resources, which are used by the Draft generation model in the Draft generation model pool to generate Drafts and / or the Draft verification model in the Draft verification model pool to execute the verification process.

[0027] In the above technical solution, the number of drafts can be adaptively adjusted based on the current inference resource status, voluntary occupancy, etc.

[0028] In combination with the first aspect, in certain implementations of the first aspect, the at least one Draft generation model includes a Draft asynchronous generation model, and the method further includes: when the inference load is less than a preset threshold, activating the Draft asynchronous generation model to generate a Draft; and storing the Draft generated by the Draft asynchronous generation model in the Draft pool.

[0029] In the above technical solution, when the inference load is less than the preset threshold, the Draft asynchronous generation model can be activated to generate Draft, avoiding the mutual waiting between the Draft generation model and the Draft verification model in the synchronous mode, thereby improving the utilization of inference resources in the model inference process, improving the efficiency of model inference, and reducing the cost of model inference.

[0030] In combination with the first aspect, in some implementations of the first aspect, the method further includes: automatically applying for / releasing the model reasoning resource based on usage of the model reasoning resource.

[0031] In the above technical solution, the inference computing resources can be applied / released in real time in combination with the current inference system status.

[0032] In combination with the first aspect, in certain implementations of the first aspect, the Draft pool includes a shared Draft pool and a private Draft pool of the first user, and a target Draft related to the text prompt information is obtained from the shared Draft pool and / or the private Draft pool of the first user.

[0033] In the above technical solution, the Draft pool supports multi-tenancy. It consists of two parts: a shared Draft pool that can be shared among multiple tenants, and a customer-specific private Draft pool, each of which is isolated from the others. When configuring their inference services, users can choose to use the shared Draft pool for cold starts.

[0034] In combination with the first aspect, in certain implementations of the first aspect, the Draft generated by the Draft synchronization generation model is stored in the first user's private Draft pool.

[0035] In the above technical solution, after the inference service is started, the system will allocate a separate draft pool for each user. Drafts generated during each user's inference service will be written to their respective private draft pools. Drafts generated during the inference service operation of different users can be maintained independently.

[0036] In combination with the first aspect, in certain implementations of the first aspect, the Draft generated by the Draft asynchronous generation model is stored in the first user's private Draft pool.

[0037] In a second aspect, a model reasoning system is provided, which includes: a Draft verification management unit, a Draft generation management unit, and at least one Draft verification model, wherein the Draft verification management unit is used to determine at least one Draft verification model from a Draft verification model pool, wherein the Draft verification model pool includes multiple Draft verification models; the Draft generation management unit is used to determine at least one Draft generation model corresponding to the at least one Draft verification model from the Draft generation model pool, wherein the Draft generation model pool includes multiple Draft generation models; the at least one Draft verification model is used to receive text prompt information input by a first user, wherein the text prompt information is used to generate text information; the Draft verification management unit is also used to obtain a target Draft related to the text prompt information from the Draft pool, wherein the Draft pool stores multiple Drafts, wherein the multiple Drafts include a Draft generated by the at least one Draft generation model; the at least one Draft verification model is also used to verify the target Draft and obtain a verification result; the at least one Draft verification model is also used to generate the text information according to the verification result and feed it back to the first user.

[0038] In combination with the second aspect, in certain implementations of the second aspect, the at least one Draft generation model includes a Draft synchronous generation model, and the at least one Draft verification model is further used to obtain Draft generation request information based on the text prompt information, and send the Draft generation request information to the Draft generation management unit through the Draft verification management form; the Draft generation management unit is also used to send the Draft generation request information to the Draft synchronous generation model; the Draft synchronous generation model is used to generate a Draft based on the Draft generation request information and send it to the Draft generation management unit; the Draft generation management unit is also used to store the Draft generated by the Draft synchronous generation model in the Draft pool.

[0039] In combination with the second aspect, in certain implementations of the second aspect, the at least one Draft generation model includes a Draft asynchronous generation model, and the Draft verification management unit is further used to activate the Draft asynchronous generation model to generate a Draft when the inference load is less than a preset threshold; the Draft asynchronous generation model is used to generate a Draft and send it to the Draft verification management unit; the Draft verification management unit is also used to store the Draft generated by the Draft asynchronous generation model in the Draft pool.

[0040] In combination with the second aspect, in certain implementations of the second aspect, the Draft pool further stores attribute information of the Draft.

[0041] In combination with the second aspect, in certain implementations of the second aspect, the Draft verification management unit is specifically used to: obtain a target Draft related to the text prompt information from the Draft pool based on the attribute information of the Draft stored in the Draft pool.

[0042] In combination with the second aspect, in certain implementations of the second aspect, the system further includes: a Draft analysis unit, configured to retrieve and / or update the Draft stored in the Draft pool based on the attribute information of the Draft stored in the Draft pool.

[0043] In combination with the second aspect, in certain implementations of the second aspect, the Draft generation management unit is further used to operate the Draft generation model included in the Draft generation model pool based on the attribute information of the Draft stored in the Draft pool, and the operations include: activation, sleep, and release.

[0044] In combination with the second aspect, in certain implementations of the second aspect, the attribute information of the Draft includes at least one of the following: the scope of the Draft, the content, the label of the corresponding Draft generation model, the number of times it is called, the number of times it is hit, and a list of text prompt information of the hit.

[0045] In combination with the second aspect, in certain implementations of the second aspect, the Draft generation request information also includes the number of Drafts generated, and the number of Drafts generated is determined based on the usage of model reasoning resources. The model reasoning resources are used for the Draft generation model in the Draft generation model pool to generate Draft and / or the Draft verification model in the Draft verification model pool to execute the verification process.

[0046] In combination with the second aspect, in some implementations of the second aspect, the system further includes: a reasoning resource analysis unit, configured to automatically apply for / release the model reasoning resources based on usage of the model reasoning resources.

[0047] In combination with the second aspect, in certain implementations of the second aspect, the Draft pool includes a shared Draft pool and a private Draft pool of the first user, and the Draft verification management unit is specifically used to: obtain a target Draft related to the text prompt information from the shared Draft pool and / or the private Draft pool of the first user.

[0048] In combination with the second aspect, in certain implementations of the second aspect, the Draft generation management unit is specifically used to: store the Draft generated by the Draft synchronization generation model in the first user's private Draft pool.

[0049] In combination with the second aspect, in certain implementations of the second aspect, the Draft generation management unit is specifically used to: store the Draft generated by the Draft asynchronous generation model in the first user's private Draft pool.

[0050] In combination with the second aspect, in certain implementations of the second aspect, the correspondence modes between the at least one Draft verification model and the at least one Draft generation model include: one-to-one, one-to-many, many-to-one, and many-to-many.

[0051] It should be understood that the beneficial effects of the second aspect and its various implementation methods are similar to the beneficial effects of the first aspect and its various implementation methods. For details, please refer to the beneficial effects of the first aspect and its various implementation methods, and no further details will be given here.

[0052] In a third aspect, a computing device is provided, comprising a processor and a memory, and optionally, an input / output interface. The processor is configured to control the input / output interface to send and receive information, the memory is configured to store a computer program, and the processor is configured to retrieve and execute the computer program from the memory, so as to perform the method of the first aspect or any possible implementation of the first aspect.

[0053] Optionally, the processor may be a general-purpose processor, which may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like; when implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory, which may be integrated into the processor or located independently of the processor.

[0054] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0055] In a fifth aspect, a chip is provided, which obtains instructions and executes the instructions to implement the method in the above-mentioned first aspect and any implementation manner of the first aspect.

[0056] Optionally, as an implementation, the chip includes a processor and a data interface, and the processor reads instructions stored in the memory through the data interface to execute the method in the above-mentioned first aspect and any implementation of the first aspect.

[0057] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored on the memory. When the instructions are executed, the processor is used to execute the method in the first aspect and any one of the implementation methods of the first aspect.

[0058] In a sixth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device, the computing device executes the method in the first aspect and any one of the implementations of the first aspect.

[0059] In a seventh aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method in the first aspect and any one of the implementations of the first aspect.

[0060] In an eighth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method as described in the first aspect and any one of the implementations of the first aspect.

[0061] By way of example, these computer-readable storages include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0062] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium.

[0063] In a ninth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to the first aspect and any one of the implementations of the first aspect.

[0064] By way of example, these computer-readable storages include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and a hard drive.

[0065] Optionally, as an implementation manner, the above-mentioned storage medium may specifically be a non-volatile storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] FIG1 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application.

[0067] FIG2 is a schematic block diagram of a model reasoning system provided in an embodiment of the present application.

[0068] FIG3 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0069] FIG4 is a schematic flowchart of a model reasoning method provided in an embodiment of the present application.

[0070] FIG5 is a schematic block diagram of a user setting a Draft generation model and a Draft verification model provided in an embodiment of the present application.

[0071] FIG6 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0072] FIG7 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0073] FIG8 is a schematic flowchart of another model reasoning method provided in an embodiment of the present application.

[0074] FIG9 is a schematic block diagram of another user-set Draft generation model and Draft verification model provided in an embodiment of the present application.

[0075] FIG10 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0076] FIG11 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0077] FIG12 is a schematic flowchart of another model reasoning method provided in an embodiment of the present application.

[0078] Figure 13 is a schematic block diagram of another user-set Draft generation model and Draft verification model provided in an embodiment of the present application.

[0079] FIG14 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0080] Figure 15 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0081] FIG16 is a schematic flowchart of another model reasoning method provided in an embodiment of the present application.

[0082] Figure 17 is a schematic block diagram of a user A setting a Draft generation model and a Draft verification model provided in an embodiment of the present application.

[0083] Figure 18 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0084] Figure 19 is a schematic block diagram of a user B setting a Draft generation model and a Draft verification model provided in an embodiment of the present application.

[0085] Figure 20 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application.

[0086] Figure 21 is a schematic block diagram of a model reasoning device 2100 provided in an embodiment of the present application.

[0087] FIG22 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application.

[0088] FIG23 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0089] FIG24 is a schematic diagram of a network connection between computing devices 1500A and 1500B provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The technical solution in this application will be described below with reference to the accompanying drawings.

[0091] This application will present various aspects, embodiments, or features around systems including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0092] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0093] In the embodiments of the present application, “corresponding” and “relevant” may sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are consistent.

[0094] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person skilled in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0095] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0096] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0097] For the convenience of description, the concepts involved in the embodiments of the present application are first explained below.

[0098] 1. Generative Model

[0099] A generative model, also known as a generative model, is a machine learning model whose goal is to generate new data that is similar to the training data. Specifically, a generative model is a method for generating data using a probabilistic model. It can be viewed as a process of drawing samples from a prior distribution, thus generating new data rather than simply classifying existing data.

[0100] 2. Autoregressive Decoding

[0101] Autoregressive decoding is an important decoding strategy in generative models. Specifically, the generative model generates tokens in the sequence one by one. The autoregressive decoding method is token by token. That is, during the autoregressive decoding process, the probability distribution of the next token is calculated based on the existing context (that is, the previously generated token). Based on this probability distribution, a token is selected as the next output. This process is recursive, meaning that each newly generated token becomes a condition for subsequent predictions. For example, in a text generation task, the first token of the sequence is generated first, and then the second token is predicted based on this token, and so on until the complete sequence is generated.

[0102] It should be understood that the above tokens can be text, characters, words, image blocks or other types of data units, depending on the application field and tasks of the model.

[0103] 3. Draft:

[0104] A draft consists of multiple tokens with contextual relationships. In other words, multiple tokens with contextual relationships are called a draft. In the context of generative models, a draft usually means the preliminary results generated by the generative model (for example, a preliminary version or an unfinished draft), which may require further revision or verification.

[0105] 4. prompt

[0106] A prompt typically refers to a piece of text or information that serves as the initial input or guide for a generative model to generate a draft. In other words, a prompt can be considered the starting point or trigger for generating a draft. By providing an appropriate prompt, the generative model can generate a draft related to the prompt content.

[0107] 5. Verify the model

[0108] The verification model verifies multiple tokens included in the generated Draft based on the input prompt and determines at least one token that passes the verification.

[0109] In recent years, with the rapid development of large models, intelligent applications built on them have been widely implemented, influencing how people generate and consume information, such as ChatGPT and Sora. A hallmark of large models is their massive parameter count. To continuously improve their application effectiveness, they are currently undergoing rapid development, focusing on increasing their parameter count. In 2023, OpenAI released ChatGPT, which boasts hundreds of billions of parameters. Just a few months later, they released GPT4, boasting trillions of parameters. The model's significantly improved conversational performance has generated significant buzz within the industry. While this massive parameter count has made large models possible, it has also become a significant barrier to their widespread adoption. One key factor is the high cost of deploying large models as a service for inference. Due to the large number of parameters, large models require extensive computing resources during inference. However, due to the limitations of the token-by-token generation method of autoregressive decoding, large models suffer from low computing resource utilization during inference. Ensuring that models fully utilize computing resources during inference, improving model inference throughput, and reducing the cost of large model inference is a challenging and important issue.

[0110] In the scheme of reasoning based on relevant generative models, in the two key stages of Draft generation and Draft verification, the Draft generation model and the Draft verification model have a strong coupling relationship in terms of collaboration and correspondence. This strong coupling relationship is solidified when the program starts, resulting in poor flexibility in the collaboration and correspondence between the Draft generation model and the Draft verification model, which reduces the efficiency of model reasoning.

[0111] In view of this, an embodiment of the present application provides a method for model reasoning, which can improve the efficiency of model reasoning.

[0112] In one possible implementation, the method provided in the embodiment of the present application can be applied to a cloud service scenario, and the method is executed by a cloud management platform in the cloud service scenario. For ease of description, the cloud service scenario is described in detail below with reference to FIG1 .

[0113] FIG1 is a schematic block diagram of a cloud scenario applicable to an embodiment of the present application. As shown in FIG1 , the cloud scenario may include: a cloud management platform 110 , the Internet 120 , and a client 130 .

[0114] As shown in Figure 1, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services. The infrastructure includes multiple cloud data centers, each of which includes multiple servers, each of which includes cloud service resources to provide corresponding cloud services to tenants.

[0115] The cloud management platform 110 can be located in a cloud data center, which can provide an access interface (such as an interface or an application program interface (API)). The tenant can operate the client 130 to remotely access the access interface to register a cloud account and password on the cloud management platform 110, and log in to the cloud management platform 110. After the cloud management platform 110 successfully authenticates the cloud account and password, the tenant can further pay to select and purchase a virtual machine with specific specifications (processor, memory, disk) on the cloud management platform 110. After the payment is successful, the cloud management platform 110 provides the remote login account and password of the purchased virtual machine, and the client 130 can remotely log in to the virtual machine, install and run the tenant's application in the virtual machine. Therefore, the tenant can create, manage, log in and operate virtual machines in the cloud data center through the cloud management platform 110. Among them, the virtual machine can also be called a cloud server (elastic compute service, ECS) or an elastic instance (different cloud service providers have different names).

[0116] It should be understood that tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative agencies, etc.

[0117] The functions of the cloud management platform 110 include, but are not limited to, a user console, computing management services, network management services, storage management services, authentication services, and image management services. The user console provides an interface or API for interacting with tenants. The computing management service is used to manage servers running virtual machines and containers, as well as bare metal servers. The network management service is used to manage network services (such as gateways and firewalls). The storage management service is used to manage storage services (such as data bucket services). The authentication service is used to manage tenant accounts and passwords. The image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 via the Internet 120 to manage the rented cloud services.

[0118] Figure 2 is a schematic block diagram of a model reasoning system provided by an embodiment of the present application. As shown in Figure 2, before the model reasoning system (also known as an efficient decoding reasoning system) starts reasoning, the system administrator needs to complete the configuration of the Draft generation model, Draft verification model, computing resources, Draft pool and other related configurations, and the model reasoning system loads the configuration information and starts the service. During the entire service period, the model reasoning system will receive external requests, and through the collaborative work of the Draft generation model, Draft verification model and Draft pool, generate text and feedback to the user. At the same time, the model reasoning system will collect and analyze a lot of operating information during the reasoning process, and send the relevant results to the system administrator and the automatic scheduling module. The system administrator can complete the system configuration update manually or automatically, and the updated configuration can be fed back to the model reasoning system in real time and take effect in real time.

[0119] Figure 3 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application. As shown in Figure 3, the model reasoning system includes: a Draft generation model pool 310, a Draft generation management unit 320, a Draft verification model pool 330, and a Draft verification management unit 340. The following describes each component of the model reasoning system in detail.

[0120] 1. Draft generates model pool 310:

[0121] The Draft generation model pool 310 includes multiple Draft generation models, for example, Draft generation model 1 to Draft generation model M. Specifically, the multiple Draft generation models can be connected to the Draft generation model pool 310 in a hot-swappable manner.

[0122] The embodiment of the present application does not specifically limit the types of Draft generation models in the Draft generation model pool 310, and its types may include but are not limited to: statistical-based natural language processing (NLP) models, traditional deep learning models (for example, recurrent neural networks (RNN), long short-term memory networks (LSTM)), pre-trained models (for example, bidirectional encoder representations from transformers (BERT), generative pre-trained transformer (GPT)), general large models (for example, Llama, Pangu), domain large models, and user-defined models.

[0123] Each Draft generation model transmits information to the Draft generation management unit 320 , including but not limited to: the label of the Draft generation model, the resources occupied by the Draft generation model inference (Draft generation), and the Draft generated by the Draft generation model.

[0124] The embodiment of the present application does not specifically limit the form of the Draft generated by the Draft generation model. The form of the Draft may include but is not limited to: N-Gram, phrases, sentences, paragraphs, etc.

[0125] 2. Draft generation management unit 320:

[0126] The Draft generation management unit 320 is used to manage the Draft generation models in the Draft generation model pool 310. Specifically, the Draft generation management unit 320 can send information to the Draft generation model pool 310 based on the current model inference system's demand for Draft and the current usage of inference resources. This information is used to control one or more Draft generation models in the Draft generation model pool 310.

[0127] In this embodiment of the present application, the information sent by the Draft generation management unit 320 to the Draft generation model pool 310 may include, but is not limited to, the Draft generation model's tag, instructions for operating the Draft generation model, and prompts for the Draft generation model to generate a Draft. Operations on the Draft generation model include, but are not limited to, activating, dormant, and releasing the Draft generation model.

[0128] 3. Draft verification model pool 330:

[0129] The Draft verification model pool 330 includes multiple Draft verification models, for example, Draft verification model 1-Draft verification model N. Specifically, the multiple Draft verification models can be connected to the Draft verification model pool 330 in a hot-swappable manner. The Draft verification model is used to receive external requests (including prompts), generate a Response (for example, generated content), and feed it back to the user.

[0130] The embodiment of the present application does not specifically limit the type of the Draft verification model in the Draft verification model pool 330, and it can be an LLM generation model of any scale.

[0131] Each Draft verification model transmits information to the Draft verification management unit 340, including but not limited to: the label of the Draft verification model, the resources occupied by the Draft verification model reasoning (verification), the current prompt, the verification results generated by the Draft generation model, and other information.

[0132] In the embodiments of the present application, there are multiple ways of corresponding between the Draft verification model and the Draft generation model, which are not specifically limited.

[0133] Here are some possible responses:

[0134] One Draft generation model corresponds to one Draft verification model;

[0135] Multiple Draft generation models correspond to one Draft verification model;

[0136] One Draft generation model corresponds to multiple Draft verification models;

[0137] Multiple Draft generation models correspond to multiple Draft verification models.

[0138] It should be noted that the corresponding methods (corresponding modes) between the above-mentioned Draft verification model and the Draft generation model can be freely switched in real time to complete dynamic combination.

[0139] 4. Draft Verification Management Unit 340

[0140] The Draft verification management unit 340 is used to manage the Draft verification models in the Draft verification model pool 330. For example, the Draft verification management unit 340 is mainly used to distribute the Draft received from the Draft generation management unit 320 to the corresponding Draft verification model.

[0141] Below, taking the model reasoning system shown in Figure 3 as an example, in combination with Figure 4, a model reasoning method provided by an embodiment of the present application is described in detail. It should be understood that the example of Figure 4 is only to help those skilled in the art understand the embodiment of the present application, and is not intended to limit the application embodiment to the specific numerical values ​​or specific scenarios illustrated in Figure 4. Those skilled in the art can obviously make various equivalent modifications or changes based on the following example given in Figure 4, and such modifications and changes also fall within the scope of the embodiment of the present application.

[0142] Figure 4 is a schematic flow chart of a model reasoning method provided by an embodiment of the present application. As shown in Figure 4, the method may include steps 410-480, which are described in detail below.

[0143] Step 410: The user completes the settings of the Draft generation model and the Draft verification model.

[0144] In an embodiment of the present application, a user can set a Draft verification model and a Draft generation model corresponding to the Draft verification model.

[0145] It should be understood that the Draft generation model corresponding to the Draft verification model may be set by the user himself, or the system may automatically recommend a Draft generation model corresponding to the Draft verification model input by the user.

[0146] For example, as shown in FIG5 , the user selects Llama 13B as the draft verification model, and selects draft generation model 1 and draft generation model 2. At this time, the model reasoning system is shown in FIG6 .

[0147] Step 420: When the model reasoning system is running, after receiving prompts, it will send them to the Draft verification model Llama 13B in the Draft verification model pool 330.

[0148] In an embodiment of the present application, during runtime, after receiving prompts, the model inference system will send them to the Draft verification model in the Draft verification model pool 330, for example, to Llama 13B in the Draft verification model pool 330.

[0149] Step 430 : The Draft verification model Llama 13B generates a Draft generation request based on the received prompts and sends it to the Draft verification management unit 340 .

[0150] In the embodiment of the present application, after receiving prompts, the Draft verification model Llama 13B generates a Draft generation request based on the received prompts and sends it to the Draft verification management unit 340.

[0151] Step 440 : The Draft verification management unit 340 generates a new Draft generation request based on the received Draft generation request, and sends the new Draft generation request to the Draft generation management unit 320 .

[0152] In an embodiment of the present application, after the Draft verification management unit 340 receives the Draft generation request sent by the Draft verification model Llama 13B, it will generate a new Draft generation request after adding information such as the Draft generation quantity to the Draft generation request, and send the new Draft generation request to the Draft generation management unit 320.

[0153] Step 450: The Draft generation management unit 320 sends a new Draft generation request to the Draft generation model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310.

[0154] In an embodiment of the present application, after the Draft generation management unit 320 receives a new Draft generation request from the Draft verification management unit 340, it can send the new Draft generation request to the Draft generation model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310, for example, the Draft generation model 1 and the Draft generation model 2 in the Draft generation model pool 310.

[0155] Step 460 : Draft generation model 1 and Draft generation model 2 each generate a Draft according to the received new Draft generation request, and feed it back to the Draft generation management unit 320 .

[0156] In the embodiment of the present application, after Draft generation model 1 and Draft generation model 2 receive a new Draft generation request sent by the Draft generation management unit 320, they each generate a Draft according to the new Draft generation request and feed back the generated Draft to the Draft generation management unit 320.

[0157] Step 470: After the Draft generation management unit 320 collects the Draft, it sends the collected Draft to the Draft verification model Llama 13B through the Draft verification management unit 340.

[0158] In an embodiment of the present application, after the Draft generation management unit 320 receives the Drafts sent by the Draft generation model 1 and the Draft generation model 2 respectively, it can send the received Drafts to the Draft verification management unit 340, and the Draft verification management unit 340 sends the received Draft to the Draft verification model Llama 13B.

[0159] Step 480: The Draft verification model Llama 13B verifies one or more Drafts corresponding to each prompt to form a new prompt.

[0160] In an embodiment of the present application, the Draft verification model Llama 13B can verify one or more Drafts corresponding to each prompt. Specifically, the Draft verification model Llama 13B can confirm at least one generated token based on one or more Drafts corresponding to each prompt, and splice the confirmed token to the corresponding prompt (received prompt) to form a new prompt.

[0161] In an embodiment of the present application, the Draft verification model Llama 13B may also send a new prompt to the Draft verification model Llama 13B again, and iteratively repeat steps 430 to 480 multiple times until the Draft verification model Llama 13B completes content generation based on the input prompts and feeds back to the user.

[0162] In the above technical solution, the correspondence between the Draft generation model and the Draft verification model is decoupled. The corresponding modes between the models can be dynamically combined and matched in any proportion, and can be adjusted in real time based on the status of the model reasoning system.

[0163] Figure 7 is a schematic block diagram of another model reasoning system provided by an embodiment of the present application. As shown in Figure 7, the model reasoning system includes: a Draft generation model pool 310, a Draft generation management unit 320, a Draft verification model pool 330, a Draft verification management unit 340, and a Draft pool 350.

[0164] It should be understood that for the description of the Draft generation model pool 310, the Draft generation management unit 320, the Draft verification model pool 330, and the Draft verification management unit 340, please refer to the description in Figure 3, which will not be repeated here. The following focuses on the detailed description of the function of the Draft pool 350.

[0165] The Draft pool 350 is used to store the Drafts generated by the Draft generation model and the attribute tag information of each Draft.

[0166] The embodiment of the present application does not specifically limit the attribute tag information of Draft. The attribute tag information of Draft includes but is not limited to: the content field corresponding to Draft, the label of the generation model corresponding to Draft, the number of times Draft is called, the number of times Draft is hit, the prompt list hit by Draft, and other information.

[0167] It should be understood that the model reasoning system shown in FIG7 is based on the model reasoning system of FIG3 with the addition of a Draft pool 350. Based on the model reasoning system of FIG7 , the Draft generation model and the Draft verification model can collaborate in the following ways:

[0168] 1) Synchronous collaboration:

[0169] The Draft verification model sends a Draft generation request, activating the corresponding Draft generation model to generate a Draft. At the same time, the generated Draft is recorded in the Draft pool 350 to facilitate reuse of subsequent models or requests.

[0170] 2) Asynchronous collaboration:

[0171] The Draft generation model generates Drafts by itself and stores them in the Draft pool 350 . The Draft verification model sends a Draft generation request and obtains relevant Drafts from the Draft pool 350 .

[0172] 3) Hybrid collaboration:

[0173] Some Draft generation models and Draft verification models collaborate synchronously, while some Draft generation models and Draft verification models collaborate asynchronously.

[0174] Below, taking the model reasoning system shown in Figure 7 as an example, in combination with Figure 8, another model reasoning method provided by an embodiment of the present application is described in detail. It should be understood that the example of Figure 8 is only to help those skilled in the art understand the embodiment of the present application, and is not intended to limit the application embodiment to the specific numerical values ​​or specific scenarios illustrated in Figure 8. Those skilled in the art can obviously make various equivalent modifications or changes based on the following example given in Figure 8, and such modifications and changes also fall within the scope of the embodiment of the present application.

[0175] Figure 8 is a schematic flow chart of another model reasoning method provided by an embodiment of the present application. As shown in Figure 8, the method may include steps 810-880, which are described in detail below.

[0176] Step 810: The user completes the settings of the Draft generation model and the Draft verification model.

[0177] In an embodiment of the present application, a user can set a Draft verification model and a Draft generation model corresponding to the Draft verification model.

[0178] For example, as shown in Figure 9, the user selects Llama 13B as the Draft verification model, selects Draft Generation Model 1 and Draft Generation Model 2 as the synchronous models, and selects Draft Generation Model 3 as the asynchronous model. At this point, the model reasoning system is shown in Figure 10.

[0179] Step 820: When the model reasoning system is running, after receiving prompts, it will send them to the Draft verification model Llama 13B in the Draft verification model pool 330.

[0180] Similar to step 420, please refer to the description in step 420 for details, which will not be repeated here.

[0181] Step 830 : The Draft verification model Llama 13B generates a Draft generation request based on the received prompts and sends it to the Draft verification management unit 340 .

[0182] Step 840 : The Draft verification management unit 340 generates a new Draft generation request based on the received Draft generation request, and sends the new Draft generation request to the Draft pool 350 .

[0183] In an embodiment of the present application, after the Draft verification management unit 340 receives the Draft generation request sent by the Draft verification model Llama 13B, it will generate a new Draft generation request after adding information such as the Draft generation quantity to the Draft generation request, and send the new Draft generation request to the Draft pool 350.

[0184] Step 850: After receiving a new Draft generation request, the Draft pool 350 obtains the corresponding Draft.

[0185] In the embodiment of the present application, after the Draft pool 350 receives a new Draft generation request sent by the Draft verification management unit 340 , it can obtain the corresponding Draft.

[0186] Specifically, the Draft pool 350 performs the following two steps simultaneously:

[0187] 1. Draft pool 350 passes the new Draft generation request to Draft generation management unit 320. Draft generation management unit 320 sends the new Draft generation request to the synchronization models corresponding to Draft verification model Llama 13B in Draft generation model pool 310, for example, Draft generation model 1 and Draft generation model 2 in Draft generation model pool 310. Draft generation model 1 and Draft generation model 2 each generate a Draft based on the received new Draft generation request and store the generated Drafts in Draft pool 350 through Draft generation management unit 320.

[0188] 2. The Draft pool 350 searches for a Draft related to the new Draft generation request in the pool.

[0189] Step 860: The Draft pool 350 sends the obtained Draft to the Draft verification model Llama 13B through the Draft verification management unit 340.

[0190] Step 870: The Draft verification model Llama 13B verifies one or more Drafts corresponding to each prompt to form a new prompt.

[0191] In an embodiment of the present application, the Draft verification model Llama 13B can verify one or more Drafts corresponding to each prompt. Specifically, the Draft verification model Llama 13B can confirm at least one generated token based on one or more Drafts corresponding to each prompt, and splice the confirmed token to the corresponding prompt (received prompt) to form a new prompt.

[0192] In an embodiment of the present application, the Draft verification model Llama 13B can also send verification information such as the hit rate of the Draft to the Draft pool 350 through the Draft verification management unit 340, and the Draft pool 350 records the verification information.

[0193] In an embodiment of the present application, the Draft verification model Llama 13B can also send the new prompt to the Draft verification model Llama 13B again, repeating steps 830-870 multiple times until the Draft verification model Llama 13B completes content generation based on the input prompts and feeds back to the user.

[0194] Step 880: When the inference service load is low, the Draft pool 350 will activate the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310 based on the collected verification information such as the hit rate of the Draft.

[0195] In an embodiment of the present application, when the inference service load is low, the Draft pool 350 generates a Draft generation request based on the collected verification information such as the hit rate of the Draft. The Draft generation management unit 320 activates the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310, for example, Draft generation model 3. The Draft generation model 3 generates a corresponding Draft based on the Draft generation request and saves it in the Draft pool 350 for use in subsequent model inference.

[0196] In this technical solution, the Draft generation model and the Draft verification model are completely decoupled. The corresponding modes and collaborative methods of the models can be dynamically combined and matched in any proportion, and can be adjusted in real time based on the state of the model reasoning system. Furthermore, the Draft pool stores the drafts generated during system operation, allowing them to be reused, which can reduce the computing power and time consumed by draft generation to a certain extent.

[0197] Figure 11 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application. As shown in Figure 11, the model reasoning system includes: a Draft generation model pool 310, a Draft generation management unit 320, a Draft verification model pool 330, a Draft verification management unit 340, a Draft pool 350, a Draft analysis unit 360, and an inference resource analysis unit 370.

[0198] It should be understood that the model reasoning system shown in Figure 11 is based on the model reasoning system of Figure 7 and adds a Draft analysis unit 360 and a reasoning resource analysis unit 370. Therefore, for the description of the Draft generation model pool 310, the Draft generation management unit 320, the Draft verification model pool 330, and the Draft verification management unit 340, please refer to the description in Figure 3. For the description of the Draft pool 350, please refer to the description in Figure 7, which will not be repeated here.

[0199] 1. Draft Analysis Unit 360:

[0200] The draft analysis unit 360 will count, analyze and manage the usage of drafts in the draft pool 350.

[0201] As an example, the main functions of the Draft analysis unit 360 include but are not limited to:

[0202] a) Analyze existing drafts in the draft pool 350 and, based on the analysis results, notify the draft generation management unit 320 to generate related drafts;

[0203] b) Analyze the prompt passed by the Draft Verification Management Unit 340 and, based on the analysis results, retrieve relevant Drafts from the Draft Pool 350 using multi-dimensional information. If relevant Drafts are missing, notify the Draft Generation Management Unit 320 to generate relevant Drafts based on the analysis results.

[0204] c) Receive draft information from the draft generation management unit 320 and verification information from the draft verification management unit 340, and based on the above information, complete operations such as screening, retrieving, modifying, deleting, and adding drafts in the draft pool 350.

[0205] 2. Reasoning resource analysis unit 370:

[0206] The reasoning resource analysis unit 370 is mainly used to analyze the current usage of reasoning resources.

[0207] As an example, the main functions of the reasoning resource analysis unit 370 include but are not limited to:

[0208] a) Receive the Draft verification model and request information sent by the Draft verification management unit 340, evaluate the current computing power and video memory available for model inference, and adaptively adjust the number of Drafts based on the current inference resource status, model resource usage, and request status;

[0209] b) receiving the Draft generation model information sent by the Draft generation management unit 320, evaluating whether the Draft generation model can be activated / dormant, and updating resource information after the Draft generation model is released;

[0210] c) Whether the currently applied inference computing resources need to be partially released, and whether additional inference computing resources need to be applied for.

[0211] Below, taking the model reasoning system shown in Figure 11 as an example, in combination with Figure 12, another model reasoning method provided by an embodiment of the present application is described in detail. It should be understood that the example of Figure 12 is only to help those skilled in the art understand the embodiment of the present application, and is not intended to limit the application embodiment to the specific numerical values ​​or specific scenarios illustrated in Figure 12. Those skilled in the art can obviously make various equivalent modifications or changes based on the following examples given in Figure 12, and such modifications and changes also fall within the scope of the embodiment of the present application.

[0212] Figure 12 is a schematic flow chart of another model reasoning method provided by an embodiment of the present application. As shown in Figure 12, the method may include steps 1210-1280, and steps 1210-1280 are described in detail below.

[0213] Step 1210: The user completes the settings of the Draft generation model and the Draft verification model.

[0214] In an embodiment of the present application, a user can set a Draft verification model and a Draft generation model corresponding to the Draft verification model.

[0215] For example, as shown in Figure 13, the user selected Llama 13B as the Draft verification model, selected synchronous Draft Generation Model 1 and Draft Generation Model 2 as the Draft generation model, and selected asynchronous Draft Generation Model 3 as the Draft generation model. Furthermore, in Figure 13, the user also enabled data adaptation and automatic scheduling for the Draft generation model. The model inference system now appears as shown in Figure 14.

[0216] Step 1220: When the model reasoning system is running, after receiving prompts, it will send them to the Draft verification model Llama 13B in the Draft verification model pool 330.

[0217] Similar to step 420, please refer to the description in step 420 for details, which will not be repeated here.

[0218] Step 1230 : The Draft verification model Llama 13B generates a Draft generation request based on the received prompts and sends it to the Draft verification management unit 340 .

[0219] Step 1240 : The Draft verification management unit 340 generates a new Draft generation request based on the received Draft generation request, and sends the new Draft generation request to the Draft analysis unit 360 .

[0220] In an embodiment of the present application, after the Draft verification management unit 340 receives the Draft generation request sent by the Draft verification model Llama 13B, it automatically sets additional information such as the Draft generation quantity for each prompt (data adaptation) based on the resource utilization and other information provided by the reasoning resource analysis unit 370, and then generates a new Draft generation request, and sends the new Draft generation request to the Draft analysis unit 360.

[0221] Step 1250: After receiving the new draft generation request, the draft analysis unit 360 obtains the corresponding draft.

[0222] In the embodiment of the present application, after the Draft analysis unit 360 receives the new Draft generation request sent by the Draft verification management unit 340, it can obtain the corresponding Draft.

[0223] Specifically, the Draft analysis unit 360 performs the following two steps simultaneously:

[0224] 1. The Draft Analysis Unit 360 passes the new Draft Generation Request to the Draft Generation Management Unit 320. The Draft Generation Management Unit 320 sends the new Draft Generation Request to the synchronization models corresponding to the Draft Verification Model Llama 13B in the Draft Generation Model Pool 310, for example, Draft Generation Model 1 and Draft Generation Model 2 in the Draft Generation Model Pool 310. Draft Generation Model 1 and Draft Generation Model 2 each generate a Draft based on the received new Draft Generation Request and send the generated Draft to the Draft Analysis Unit 360 via the Draft Generation Management Unit 320.

[0225] 2. The Draft analysis unit 360 generates multi-dimensional analysis information of prompts from multiple perspectives such as domain and semantics, and searches the Draft pool 350 for a Draft related to the new Draft generation request based on the multi-dimensional analysis information of prompts.

[0226] Step 1260: The Draft analysis unit 360 sends the obtained Draft to the Draft verification model Llama 13B through the Draft verification management unit 340.

[0227] Step 1270: The Draft verification model Llama 13B verifies one or more Drafts corresponding to each prompt to form a new prompt.

[0228] In an embodiment of the present application, the Draft verification model Llama 13B can verify one or more Drafts corresponding to each prompt. Specifically, the Draft verification model Llama 13B can confirm at least one generated token based on one or more Drafts corresponding to each prompt, and splice the confirmed token to the corresponding prompt (received prompt) to form a new prompt.

[0229] In the embodiment of the present application, the Draft verification model Llama 13B can also send verification information such as the hit rate of the Draft to the Draft analysis unit 360 through the Draft verification management unit 340, and the Draft analysis unit 360 records the verification information.

[0230] In an embodiment of the present application, the Draft verification model Llama 13B can also send the new prompt to the Draft verification model Llama 13B again, repeating steps 1230-1270 multiple times until the Draft verification model Llama 13B completes content generation based on the input prompts and feeds back to the user.

[0231] Step 1280: When the inference service load is low, the Draft analysis unit 360 activates the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310 based on the collected verification information such as the hit rate of the Draft.

[0232] In an embodiment of the present application, when the inference service load is low, the Draft analysis unit 360 will generate a Draft generation request based on the collected verification information such as the hit rate of the Draft. The Draft generation management unit 320 will combine the inference resource utilization and other information provided by the current inference resource analysis unit 370 to activate the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310, for example, Draft generation model 3. The Draft generation model 3 will generate a corresponding Draft based on the Draft generation request and save it in the Draft pool 350 for use in subsequent model inference.

[0233] In the embodiment of the present application, if the Draft generation management unit 320 requires activation of the Draft generation model 3 and there are currently insufficient reasoning resources, the reasoning resource analysis unit 370 will provide new reasoning resources for the Draft generation model 3 to generate a Draft.

[0234] In an embodiment of the present application, if the reasoning resource utilization information provided by the reasoning resource analysis unit 370 indicates that there are currently unused reasoning resources and the Draft generation model 3 is not activated, the reasoning resource analysis unit 370 will automatically release the reasoning resources.

[0235] In the above technical solution, on the one hand, the number of Drafts generated can be adaptively adjusted based on the current state of resources; on the other hand, the model reasoning system can complete self-adjustment based on different states, including adjusting the correspondence and coordination between the Draft generation model and the Draft verification model, self-maintenance of the Draft pool, application and release of reasoning resources, etc.; on the other hand, since the Draft analysis unit 360 generates multi-dimensional analysis information of prompts from multiple perspectives such as domain and semantics, and searches for Drafts related to the Draft generation request in the Draft pool 350 based on the multi-dimensional analysis information of the prompts, the Draft recall is more accurate, which brings a higher Draft hit rate in the verification stage.

[0236] Figure 15 is a schematic block diagram of another model reasoning system provided in an embodiment of the present application. As shown in Figure 15, the model reasoning system includes: a Draft generation model pool 310, a Draft generation management unit 320, a Draft verification model pool 330, a Draft verification management unit 340, a Draft pool 350, a Draft analysis unit 360, and an inference resource analysis unit 370.

[0237] It should be understood that the model reasoning system shown in Figure 15 is based on the model reasoning system shown in Figure 11, so that the Draft pool 350 supports multi-tenancy mode. The Draft pool 350 is mainly composed of two parts, one is a preset Draft pool (also called a shared Draft pool), which can be shared among multiple tenants, and the other is the customer's private Draft pool. Each customer's Draft pool is isolated from each other. When different users configure their own reasoning services, they can choose whether to use the shared Draft pool for cold start. After the service is started, the system will allocate an independent Draft pool for each user, and the Draft generated during the reasoning service process of each user will be written into their respective Draft pools.

[0238] Below, taking the model reasoning system shown in Figure 15 as an example, in combination with Figure 16, another model reasoning method provided by an embodiment of the present application is described in detail. It should be understood that the example of Figure 16 is only to help those skilled in the art understand the embodiment of the present application, and is not intended to limit the application embodiment to the specific numerical values ​​or specific scenarios illustrated in Figure 16. Those skilled in the art can obviously make various equivalent modifications or changes based on the following examples given in Figure 16, and such modifications and changes also fall within the scope of the embodiment of the present application.

[0239] Figure 16 is a schematic flow chart of a model reasoning method provided by an embodiment of the present application. As shown in Figure 16, the method may include steps 1610-1680, and steps 1610-1680 are described in detail below.

[0240] Step 1610: User A completes the settings of the Draft generation model and the Draft verification model.

[0241] For example, as shown in Figure 17, user A selects Llama 13B as the Draft verification model. Among the selected Draft generation models, the synchronous models are Draft Generation Model 1 and Draft Generation Model 2, and the asynchronous model is Draft Generation Model 3. Furthermore, in Figure 17, user A also chooses to enable the data adaptation function, the automatic scheduling function of the Draft generation model, and the use of the shared Draft pool in Draft pool 350. At this point, the model inference system is shown in Figure 18, where Draft pool 350 in Figure 18 includes the shared Draft pool and user A's private Draft pool.

[0242] Step 1620: When the model reasoning system is running, after receiving prompts, it will send them to the Draft verification model Llama 13B in the Draft verification model pool 330.

[0243] Similar to step 420, please refer to the description in step 420 for details, which will not be repeated here.

[0244] Step 1630 : The Draft verification model Llama 13B generates a Draft generation request based on the received prompts and sends it to the Draft verification management unit 340 .

[0245] Step 1640 : The Draft verification management unit 340 generates a new Draft generation request based on the received Draft generation request, and sends the new Draft generation request to the Draft analysis unit 360 .

[0246] Step 1650: After receiving the new draft generation request, the draft analysis unit 360 obtains the corresponding draft.

[0247] In the embodiment of the present application, after the Draft analysis unit 360 receives the new Draft generation request sent by the Draft verification management unit 340, it can obtain the corresponding Draft.

[0248] Specifically, the Draft analysis unit 360 performs the following three steps simultaneously:

[0249] 1. The Draft Analysis Unit 360 passes the new Draft Generation Request to the Draft Generation Management Unit 320. The Draft Generation Management Unit 320 sends the new Draft Generation Request to the synchronization models corresponding to the Draft Verification Model Llama 13B in the Draft Generation Model Pool 310, for example, Draft Generation Model 1 and Draft Generation Model 2 in the Draft Generation Model Pool 310. Draft Generation Model 1 and Draft Generation Model 2 each generate a Draft based on the received new Draft Generation Request and send the generated Draft to the Draft Analysis Unit 360 via the Draft Generation Management Unit 320.

[0250] 2. Draft analysis unit 360 generates multi-dimensional analysis information of prompts from multiple perspectives, such as domain and semantics, and searches for a draft related to the new draft generation request in the shared draft pool in Draft pool 350 based on the multi-dimensional analysis information of prompts;

[0251] 3. The Draft analysis unit 360 generates multi-dimensional analysis information of prompts from multiple perspectives such as domain and semantics, and searches for a Draft related to the new Draft generation request in the private Draft pool of user A in the Draft pool 350 based on the multi-dimensional analysis information of prompts.

[0252] Step 1660: The Draft analysis unit 360 sends the obtained Draft to the Draft verification model Llama 13B through the Draft verification management unit 340.

[0253] Step 1670: The Draft verification model Llama 13B verifies one or more Drafts corresponding to each prompt to form a new prompt.

[0254] In an embodiment of the present application, the Draft verification model Llama 13B can verify one or more Drafts corresponding to each prompt. Specifically, the Draft verification model Llama 13B can confirm at least one generated token based on one or more Drafts corresponding to each prompt, and splice the confirmed token to the corresponding prompt (received prompt) to form a new prompt.

[0255] In this embodiment of the present application, the draft verification model Llama 13B may also send verification information such as the draft's hit rate to the draft analysis unit 360 via the draft verification management unit 340, and the draft analysis unit 360 will record the verification information. The draft analysis unit 360 will analyze the currently obtained draft based on the verification information and complete operations on the relevant drafts in user A's private draft pool based on the analysis results.

[0256] In an embodiment of the present application, the Draft verification model Llama 13B can also send the new prompt to the Draft verification model Llama 13B again, repeating steps 1630-1670 multiple times until the Draft verification model Llama 13B completes content generation based on the input prompts and feeds back to user A.

[0257] Step 1680: When the inference service load is low, the Draft analysis unit 360 activates the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310 based on the collected verification information such as the hit rate of the Draft.

[0258] In an embodiment of the present application, when the inference service load is low, the Draft analysis unit 360 will form a Draft generation request based on the collected verification information such as the hit rate of the Draft. The Draft generation management unit 320 will combine the inference resource utilization and other information provided by the current inference resource analysis unit 370 to activate the synchronization model corresponding to the Draft verification model Llama 13B in the Draft generation model pool 310, for example, Draft generation model 3. The Draft generation model 3 will generate the corresponding Draft based on the Draft generation request and save it in the private Draft pool of user A in the Draft pool 350 for use in subsequent model inference.

[0259] Similarly, if the Draft Generation Management Unit 320 requests the activation of the Draft Generation Model 3 and there are currently insufficient reasoning resources, the reasoning resource analysis unit 370 will allocate new reasoning resources for the Draft Generation Model 3 to generate a draft. If the reasoning resource utilization information provided by the reasoning resource analysis unit 370 indicates that there are currently unused reasoning resources and the Draft Generation Model 3 is not activated, the reasoning resource analysis unit 370 will automatically release the reasoning resources.

[0260] Optionally, in some embodiments, it is assumed that user B has also completed the system configuration. For example, as shown in Figure 19, the Draft verification model selected by user B is Llama 7B, and among the selected Draft generation models, the synchronous models are Draft generation model 1 and Draft generation model 4, and the asynchronous model is Draft generation model 5. In addition, in Figure 19, user B also chooses to enable the data adaptation function, enable the automatic scheduling function of the Draft generation model, and enable the use of the shared Draft pool in the Draft pool 350. At this time, the model reasoning system is shown in Figure 20, and the Draft pool 350 in Figure 20 includes the shared Draft pool, user A's private Draft pool, and user B's private Draft pool.

[0261] It should be noted that user B can also complete the model inference process according to steps 1610-1680 performed by user A. The difference is that user B's Draft verification model is Llama 7B, the synchronous models are Draft Generation Model 1 and Draft Generation Model 4, of which Draft Generation Model 1 is shared with user A, and the asynchronous model is Draft Generation Model 5. During the inference process of user B, the Draft analysis unit 360 can read the shared Draft pool in Draft pool 350, but can only perform read and write operations on user B's private Draft pool in Draft pool 350.

[0262] In the above technical solution, the shared draft pool in the draft pool can be shared by multiple tenants to help users complete inference acceleration during cold start. The private draft pools of different users in the draft pool are independent of each other, so that the drafts generated by different users during the operation of the inference service can be independently maintained.

[0263] The above description, in conjunction with Figures 1 to 20 , details the method for providing model reasoning according to an embodiment of the present application. The following description, in conjunction with Figures 21 to 24 , details the embodiment of the apparatus according to the present application. It should be understood that the description of the method embodiment corresponds to the description of the apparatus embodiment. Therefore, for portions not described in detail, reference can be made to the preceding method embodiment.

[0264] Figure 21 is a schematic block diagram of a model reasoning device 2100 provided in an embodiment of the present application. The device 2100 can be implemented by software, hardware, or a combination of both. The device 2100 provided in an embodiment of the present application can implement the method flow shown in the embodiment of the present application, and the device 2100 includes: a receiving module 2110, a processing module 2120, and a feedback module 2130, wherein the receiving module 2110 is used to receive text prompt information input by the first user, and the text prompt information is used to generate text information; the processing module 2120 is used to determine at least one Draft verification model from the Draft verification model pool, and the Draft verification model pool includes multiple Draft verification models; determine at least one Draft generation model corresponding to the at least one Draft verification model from the Draft generation model pool, wherein the Draft generation model pool includes multiple Draft generation models, and the at least one Draft generation model includes Dr. aft synchronous generation model and Draft asynchronous generation model; obtain Draft generation request information according to the text prompt information, send the Draft generation request information to the Draft synchronous generation model, the Draft synchronous generation model is used to generate a Draft according to the Draft generation request information; and / or obtain a Draft related to the text prompt information from the Draft pool, the Draft pool includes the Draft generated by the Draft asynchronous generation model; verify the Draft generated by the Draft synchronous generation model and / or the Draft obtained from the Draft pool through the at least one Draft verification model to obtain a verification result; the feedback module 2130 is used to generate the text information according to the verification result and feed it back to the first user.

[0265] Optionally, the correspondence modes between the at least one Draft verification model and the at least one Draft generation model include: one-to-one, one-to-many, many-to-one, and many-to-many.

[0266] Optionally, the device 2100 further includes: a storage module, configured to store the Draft generated by the Draft synchronization generation model in the Draft pool.

[0267] Optionally, the Draft pool also stores Draft attribute information.

[0268] Optionally, the processing module 2120 is specifically used to: obtain a Draft related to the text prompt information from the Draft pool according to the attribute information of the Draft stored in the Draft pool.

[0269] Optionally, the processing module 2120 is further configured to retrieve and / or update the Draft stored in the Draft pool based on the attribute information of the Draft stored in the Draft pool.

[0270] Optionally, the processing module 2120 is further configured to operate the Draft generation model included in the Draft generation model pool according to the attribute information of the Draft stored in the Draft pool, and the operation includes: activation, dormancy, and release.

[0271] Optionally, the attribute information of the Draft includes at least one of the following: the scope of the Draft, the content, the label of the corresponding Draft generation model, the number of times it is called, the number of times it is hit, and a list of text prompt information of the hits.

[0272] Optionally, the Draft generation request information also includes the number of generated Drafts, which is determined based on the usage of model reasoning resources. The model reasoning resources are used by the Draft generation model in the Draft generation model pool to generate Drafts and / or the Draft verification model in the Draft verification model pool to perform the verification process.

[0273] Optionally, the processing module 2120 is further used to activate the Draft asynchronous generation model to generate a Draft when the inference load is less than a preset threshold; the storage module is further used to store the Draft generated by the Draft asynchronous generation model in the Draft pool.

[0274] Optionally, the processing module 2120 is further configured to automatically apply for / release the model inference resource based on usage of the model inference resource.

[0275] Optionally, the Draft pool includes a shared Draft pool and a private Draft pool of the first user, and the processing module 2120 is specifically used to obtain the Draft related to the text prompt information from the shared Draft pool and / or the private Draft pool of the first user.

[0276] Optionally, the storage module is specifically used to: store the Draft generated by the Draft synchronization generation model in the first user's private Draft pool.

[0277] Optionally, the storage module is further used to store the Draft generated by the Draft asynchronous generation model in the first user's private Draft pool.

[0278] The device 2100 herein may be embodied in the form of a functional module. The term "module" herein may be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0279] For example, a "module" can be a software program, hardware circuit, or a combination of the two that implements the aforementioned functions. For example, the implementation of receiving module 2110 will be described below, using receiving module 2110 as an example. Similarly, the implementation of other modules, such as processing module 2120, feedback module 2130, and storage module, can refer to the implementation of receiving module 2110.

[0280] The receiving module 2110 is taken as an example of a software functional unit, and the receiving module 2110 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the receiving module 2110 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Generally, a region may include multiple AZs.

[0281] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0282] As an example of a hardware functional unit, receiving module 2110 may include at least one computing device, such as a server. Alternatively, receiving module 2110 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0283] The multiple computing devices included in the receiving module 2110 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 2110 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 2110 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0284] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0285] It should be noted that: when the device provided in the above embodiment executes the above method, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the receiving module 2110 can be used to execute any step in the above method, the processing module 2120 can be used to execute any step in the above method, the feedback module 2130 can be used to execute any step in the above method, and the storage module can be used to execute any step in the above method. The steps that the receiving module 2110, the processing module 2120, the feedback module 2130, and the storage module are responsible for implementing can be specified as needed, and the full functions of the above device can be realized by respectively implementing different steps in the above method through the receiving module 2110, the processing module 2120, the feedback module 2130, and the storage module.

[0286] In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments above and will not be repeated here.

[0287] The method provided in the embodiment of the present application can be performed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory and a memory control unit, and the function and structure of the hardware are subsequently described in detail. The operating system is any one or more computer operating systems that implement business processing through a process, for example, a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system or a Windows operating system. The application layer includes applications such as a browser, an address book, a word processing software, and an instant messaging software. Furthermore, optionally, the computer system is a handheld device such as a smart phone, or a terminal device such as a personal computer, and this application is not particularly limited, as long as the method provided in the embodiment of the present application can be used. The execution subject of the method provided in the embodiment of the present application can be a computing device, or a functional module in a computing device that can call a program and execute a program.

[0288] The following is a detailed description of a computing device provided in an embodiment of the present application in conjunction with Figure 22.

[0289] FIG22 is a schematic diagram of the architecture of a computing device 1500 provided in an embodiment of the present application. The computing device 1500 may be a server, a computer, or other device with computing capabilities. The computing device 1500 shown in FIG22 includes: at least one processor 1510 and a memory 1520.

[0290] It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0291] The processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the method provided in this application. Alternatively, the processor 1510 executes the instructions in the memory 1520 so that the computing device 1500 implements the functional modules provided in this application, thereby implementing the method provided in this application.

[0292] Optionally, the computing device 1500 further includes a communication interface 1530. The communication interface 1530 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0293] Optionally, the computing device 1500 further includes a system bus 1540, wherein the processor 1510, the memory 1520, and the communication interface 1530 are respectively connected to the system bus 1540. The processor 1510 can access the memory 1520 through the system bus 1540. For example, the processor 1510 can read and write data or execute code in the memory 1520 through the system bus 1540. The system bus 1540 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1540 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 22, but it does not mean that there is only one bus or one type of bus.

[0294] In one possible implementation, the processor 1510 is primarily responsible for interpreting computer program instructions (or codes) and processing data in the computer software. The computer program instructions and the data in the computer software may be stored in the memory 1520 or the cache 1516.

[0295] Optionally, the processor 1510 may be an integrated circuit chip having signal processing capabilities. By way of example and not limitation, the processor 1510 is a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor is a microprocessor, etc. For example, the processor 1510 is a central processing unit (CPU).

[0296] Optionally, each processor 1510 includes at least one processing unit 1512 and a memory control unit 1514 .

[0297] Processing unit 1512, also known as a core, is the most important component of a processor. Processing unit 1512 is manufactured from single-crystal silicon using a specific production process. All processor calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, leveraging parallel computing capabilities to accelerate program execution. Each processing unit has a fixed logical structure. For example, a processing unit includes logical units such as a level 1 cache, a level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0298] In one implementation example, the memory control unit 1514 is configured to control data exchange between the memory 1520 and the processing unit 1512. Specifically, the memory control unit 1514 receives memory access requests from the processing unit 1512 and controls access to the memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0299] In one implementation example, each memory control unit 1514 addresses the memory 1520 via the system bus. An arbiter (not shown in FIG. 22 ) is configured in the system bus to handle and coordinate competing accesses by multiple processing units 1512 .

[0300] In an implementation example, the processing unit 1512 and the memory control unit 1514 are connected to each other via connection lines inside the chip, such as address lines, so as to achieve communication between the processing unit 1512 and the memory control unit 1514.

[0301] Optionally, each processor 1510 also includes a cache 1516, which is a buffer for data exchange (called a cache). When a processing unit 1512 needs to read data, it first searches the cache for the required data. If the data is found, it executes the request directly; if not, it searches the memory. Because the cache runs much faster than the memory, the cache helps the processing unit 1512 run faster.

[0302] Memory 1520 can provide runtime space for processes in computing device 1500. For example, memory 1520 stores computer programs (specifically, program code) used to generate processes. After the computer program is executed by the processor to generate a process, the processor allocates corresponding storage space for the process in memory 1520. Furthermore, the aforementioned storage space further includes a text segment, an initialized data segment, a bit-initialized data segment, a stack segment, a heap segment, and the like. Memory 1520 stores data generated during the execution of the aforementioned processes, such as intermediate data or process data, in the storage space corresponding to the aforementioned processes.

[0303] Optionally, the memory is also called the internal memory. Its function is to temporarily store the data processed by the processor 1510 and the data exchanged with external storage such as a hard disk. As long as the computer is running, the processor 1510 will load the data to be calculated into the internal memory for calculation. When the calculation is completed, the processing unit 1512 will transmit the result.

[0304] By way of example and not limitation, memory 1520 is a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory 1520 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0305] The structure of the computing device 1500 listed above is only an example, and the present application is not limited thereto. The computing device 1500 of the embodiment of the present application includes various hardware in the computer system in the prior art. For example, the computing device 1500 also includes other memories in addition to the memory 1520, such as disk storage, etc. Those skilled in the art should understand that the computing device 1500 may also include other devices necessary to achieve normal operation. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned computing device 1500 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned computing device 1500 may also include only the devices necessary to implement the embodiment of the present application, and does not necessarily include all the devices shown in Figure 22.

[0306] The present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0307] As shown in Figure 23, the computing device cluster includes at least one computing device 1500. The memory 1520 in one or more computing devices 1500 in the computing device cluster may store the same instructions for executing the above method.

[0308] In some possible implementations, the memory 1520 of one or more computing devices 1500 in the computing device cluster may also store some instructions for executing the above method. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions of the above method.

[0309] It should be noted that the memory 1520 in different computing devices 1500 in the computing device cluster can store different instructions, each for performing part of the functions of the above-mentioned apparatus. In other words, the instructions stored in the memory 1520 in different computing devices 1500 can implement the functions of one or more modules in the above-mentioned apparatus.

[0310] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), among others. FIG. 24 illustrates one possible implementation. As shown in FIG. 24 , two computing devices 1500A and 1500B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device.

[0311] It should be understood that the functionality of the computing device 1500A shown in FIG24 may also be implemented by multiple computing devices 1500. Similarly, the functionality of the computing device 1500B may also be implemented by multiple computing devices 1500.

[0312] This embodiment also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the method provided above, or causes the computing device to implement the functions of the apparatus provided above.

[0313] This embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of storing data on a computing device, or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.

[0314] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0315] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0316] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0317] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0318] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0319] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0320] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0321] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for model reasoning, characterized in that: The method comprises: Determine at least one Draft verification model from a Draft verification model pool, wherein the Draft verification model pool includes multiple Draft verification models; Determining at least one Draft generation model corresponding to the at least one Draft verification model from a Draft generation model pool, wherein the Draft generation model pool includes a plurality of Draft generation models; receiving text prompt information input by a first user, wherein the text prompt information is used to generate text information; Acquire a target draft related to the text prompt information from a draft pool, wherein the draft pool stores a plurality of drafts, the plurality of drafts including a draft generated by the at least one draft generation model; Verifying the target Draft using the at least one Draft verification model and obtaining a verification result; The text information is generated according to the verification result and fed back to the first user.

2. The method according to claim 1, characterized in that The at least one Draft generation model includes a Draft synchronization generation model, and the method further includes: Obtaining Draft generation request information according to the text prompt information; Sending the Draft generation request information to the Draft synchronization generation model, wherein the Draft synchronization generation model is used to generate a Draft according to the Draft generation request information; The Draft generated by the Draft synchronization generation model is stored in the Draft pool.

3. The method according to claim 1 or 2, characterized in that The at least one Draft generation model includes a Draft asynchronous generation model, and the method further includes: When the inference load is less than a preset threshold, activating the Draft asynchronous generation model to generate Draft; The Draft generated by the Draft asynchronous generation model is stored in the Draft pool.

4. The method according to any one of claims 1 to 3, characterized in that The Draft pool also stores Draft attribute information.

5. The method according to claim 4, characterized in that The step of obtaining a target Draft related to the text prompt information from the Draft pool includes: According to the attribute information of the Draft stored in the Draft pool, a target Draft related to the text prompt information is obtained from the Draft pool.

6. The method according to claim 4 or 5, characterized in that The method further comprises: According to the attribute information of the Draft stored in the Draft pool, the Draft stored in the Draft pool is retrieved and / or updated.

7. The method according to any one of claims 4 to 6, characterized in that The method further comprises: According to the attribute information of the Draft stored in the Draft pool, operations are performed on the Draft generation models included in the Draft generation model pool, and the operations include: activation, dormancy, and release.

8. The method according to any one of claims 4 to 7, characterized in that The attribute information of the Draft includes at least one of the following: the domain of the Draft, the content, the label of the corresponding Draft generation model, the number of times it is called, the number of times it is hit, and a list of text prompt information of the hit.

9. The method according to any one of claims 1 to 8, characterized in that The Draft generation request information also includes the number of Drafts generated, which is determined based on the usage of model reasoning resources. The model reasoning resources are used by the Draft generation model in the Draft generation model pool to generate Drafts and / or the Draft verification model in the Draft verification model pool to perform the verification process.

10. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Automatically apply for / release the model reasoning resources based on their usage.

11. The method according to any one of claims 1 to 10, characterized in that The Draft pool includes a shared Draft pool and a private Draft pool of the first user. The step of obtaining a target Draft related to the text prompt information from the Draft pool includes: Obtain a target Draft related to the text prompt information from the shared Draft pool and / or the first user's private Draft pool.

12. The method according to any one of claims 2 to 11, characterized in that The storing the Draft generated by the Draft synchronization generation model in the Draft pool includes: The Draft generated by the Draft synchronization generation model is stored in the first user's private Draft pool.

13. The method according to any one of claims 3 to 12, characterized in that The storing the Draft generated by the Draft asynchronous generation model in the Draft pool includes: The Draft generated by the Draft asynchronous generation model is stored in the first user's private Draft pool.

14. The method according to any one of claims 1 to 13, characterized in that The correspondence modes between the at least one Draft verification model and the at least one Draft generation model include: one-to-one, one-to-many, many-to-one, and many-to-many.

15. A model reasoning system, characterized in that: The system comprises: A Draft verification management unit, configured to determine at least one Draft verification model from a Draft verification model pool, wherein the Draft verification model pool includes a plurality of Draft verification models; a Draft generation management unit, configured to determine at least one Draft generation model corresponding to the at least one Draft verification model from a Draft generation model pool, wherein the Draft generation model pool includes a plurality of Draft generation models; The at least one Draft verification model is configured to receive text prompt information input by a first user, wherein the text prompt information is used to generate text information; The Draft verification management unit is further configured to obtain a target Draft related to the text prompt information from a Draft pool, wherein the Draft pool stores a plurality of Drafts, including a Draft generated by the at least one Draft generation model; The at least one Draft verification model is further used to verify the target Draft and obtain a verification result; The at least one Draft verification model is further used to generate the text information according to the verification result and feed it back to the first user.

16. The system according to claim 15, characterized in that The at least one Draft generation model includes a Draft synchronization generation model, The at least one Draft verification model is further configured to obtain Draft generation request information according to the text prompt information, and send the Draft generation request information to the Draft generation management unit via the Draft verification management form; The Draft generation management unit is further configured to send the Draft generation request information to the Draft synchronization generation model; The Draft synchronization generation model is used to generate a Draft according to the Draft generation request information and send it to the Draft generation management unit; The Draft generation management unit is further configured to store the Draft generated by the Draft synchronization generation model in the Draft pool.

17. The system according to claim 15 or 16, characterized in that The at least one Draft generation model includes a Draft asynchronous generation model, The Draft verification management unit is further configured to activate the Draft asynchronous generation model to generate a Draft when the inference load is less than a preset threshold; The Draft asynchronous generation model is used to generate a Draft and send it to the Draft verification management unit; The Draft verification management unit is further used to store the Draft generated by the Draft asynchronous generation model in the Draft pool.

18. The system according to any one of claims 15 to 17, characterized in that The Draft pool also stores Draft attribute information.

19. The system according to claim 18, wherein: The Draft verification management unit is specifically used to: According to the attribute information of the Draft stored in the Draft pool, a target Draft related to the text prompt information is obtained from the Draft pool.

20. The system according to claim 18 or 19, characterized in that The system further comprises: The Draft analysis unit is used to retrieve and / or update the Draft stored in the Draft pool based on the attribute information of the Draft stored in the Draft pool.

21. The system according to any one of claims 18 to 20, characterized in that The Draft generation management unit is further used to operate the Draft generation model included in the Draft generation model pool according to the attribute information of the Draft stored in the Draft pool, and the operation includes: activation, dormancy, and release.

22. The system according to any one of claims 18 to 21, characterized in that The attribute information of the Draft includes at least one of the following: the domain of the Draft, the content, the label of the corresponding Draft generation model, the number of times it is called, the number of times it is hit, and a list of text prompt information of the hit.

23. The system according to any one of claims 15 to 21, characterized in that The Draft generation request information also includes the number of Drafts generated, which is determined based on the usage of model reasoning resources. The model reasoning resources are used by the Draft generation model in the Draft generation model pool to generate Drafts and / or the Draft verification model in the Draft verification model pool to perform the verification process.

24. The system according to any one of claims 15 to 23, characterized in that The system further comprises: The reasoning resource analysis unit is used to automatically apply for / release the model reasoning resources according to the usage of the model reasoning resources.

25. The system according to any one of claims 15 to 24, characterized in that The Draft pool includes a shared Draft pool and a private Draft pool of the first user. The Draft verification management unit is specifically configured to: Obtain a target Draft related to the text prompt information from the shared Draft pool and / or the first user's private Draft pool.

26. The system according to any one of claims 16 to 25, characterized in that The Draft generation management unit is specifically used to: The Draft generated by the Draft synchronization generation model is stored in the first user's private Draft pool.

27. The system according to any one of claims 17 to 26, characterized in that The Draft generation management unit is specifically used to: The Draft generated by the Draft asynchronous generation model is stored in the first user's private Draft pool.

28. The system according to any one of claims 15 to 27, characterized in that The correspondence modes between the at least one Draft verification model and the at least one Draft generation model include: one-to-one, one-to-many, many-to-one, and many-to-many.

29. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 14.

30. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 14.

31. A computer-readable storage medium, characterized in that The method comprises computer program instructions which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 14.

Images