Inference scheduling method and device, computer equipment, computer readable storage medium and computer program product

Abstract

The invention provides a reasoning scheduling method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of receiving a reasoning request of a first user, judging whether available computing power resources in first computing power resources subscribed by the first user can meet computing power resources required by the reasoning request or not, and obtaining a judgment result; under the condition that the judgment result shows that the first computing power resource can meet the computing power resource required by the reasoning request, updating a task queue set based on the reasoning request; the task queue set at least comprises a first task queue related to a reasoning result of the reasoning request; executing a reasoning task in the reasoning request by using the first computing power resource to obtain a first reasoning result, and caching the first reasoning result in a first task queue; and obtaining the first reasoning result from the first task queue, and sending the first reasoning result to the first user, so that the overall utilization rate of the cloud computing resource pool can be improved.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a reasoning scheduling method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology across various fields, cloud providers have emerged to promote the large-scale deployment of AI capabilities. Cloud providers pool AI resources (such as GPUs and NPUs) through virtualization technology and provide services to users via the internet on an on-demand rental basis. This allows users to quickly deploy applications and systems without building and maintaining their own physical infrastructure, providing full-stack support for the research, deployment, and application of AI technology, and helping enterprises and developers efficiently build AI capabilities.

[0003] Currently, the large-scale model industry is experiencing strong growth, driving the rapid development of AI resources from cloud vendors. However, the current cloud environment faces challenges such as a shortage of AI computing resources, low performance utilization, and wasted resources in tenant clusters, severely impacting the development of AI capabilities. Taking large language models (LLMs) powered by cloud vendor AI resources as an example, a key issue is how to build a unified LLM inference scheduling layer from a global perspective of cloud resources to achieve unified scheduling of multi-tenant, multi-cluster, multi-model, and high-concurrency inference requests in the cloud, thereby improving the overall utilization of the cloud computing resource pool. Summary of the Invention

[0004] This application provides an inference scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the overall utilization rate of cloud computing resource pools.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides an inference scheduling method, the method comprising: receiving an inference request from a first user, determining whether the available computing resources in the first computing resources subscribed by the first user can meet the computing resources required by the inference request, and obtaining a determination result; If the determination result indicates that the first computing power resource can meet the computing power resource required for the inference request, the task queue set is updated based on the inference request; wherein, the task queue set includes at least a first task queue related to the inference result of the inference request; The first computing power resource is used to execute the reasoning task in the reasoning request to obtain a first reasoning result, which is cached in the first task queue. The first inference result is obtained from the first task queue and sent to the first user.

[0006] In the above scheme, the first computing power resource is represented by the number of tokens processed within the subscription period; before receiving the inference request from the first user, the method further includes: obtaining the total number of tokens corresponding to the first computing power resource within a preset period and the number of tokens consumed by the first computing power resource; and determining the number of available tokens corresponding to the first computing power resource within the preset period based on the total number of tokens and the number of tokens consumed.

[0007] In the above scheme, the first computing power resource is represented by the number of tokens processed within the subscription period; the step of determining whether the available computing power resources in the first computing power resource subscribed by the first user can meet the computing power resources required for the inference request includes: within a preset period, if the number of available tokens corresponding to the first computing power resource is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold, determining that the first computing power resource can meet the computing power resources required for the inference request; within the preset period, if the number of available tokens corresponding to the first computing power resource is less than the first threshold, and / or, the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is less than the second threshold, determining that the first computing power resource cannot meet the computing power resources required for the inference request.

[0008] In the above scheme, the inference request includes: a first user identifier corresponding to the first user, a first model identifier corresponding to the first computing resource, a first request identifier corresponding to the inference request, and inference request information including the inference task. The task queue set also includes a second task queue related to the model and a third task queue related to the first user. Updating the task queue set based on the inference request includes: constructing a first task queue associated with the first request identifier, the first task queue being used to cache the inference results obtained from executing the inference task; caching the first user identifier in the second task queue associated with the first model identifier; caching the first request identifier in the third task queue associated with the first user identifier; and generating a first mapping table based on the first request identifier and the inference request information.

[0009] In the above scheme, before using the first computing power resource to execute the inference task in the inference request, the method further includes: determining whether the number of tasks being processed by the first computing power resource has reached a preset maximum number of tasks; if it is determined that the preset maximum number of tasks has not been reached, obtaining the first request identifier from the third task queue based on the first user identifier, and obtaining the inference task from the first mapping table based on the first request identifier.

[0010] In the above scheme, the task queue set further includes a second task queue related to the model and a third task queue related to the user. The inference request includes a first request identifier corresponding to the inference request. When the judgment result indicates that the first computing power resources cannot meet the computing power resources required by the inference request, the method further includes: determining one or more other user identifiers besides the first user identifier corresponding to the first user based on the second task queue; judging whether the available computing power resources in the second computing power resources subscribed by each other user can meet the computing power resources required by the inference request based on each other user identifier; when the available computing power resources in the second computing power resources subscribed by the second user can meet the computing power resources required by the inference request, caching the first request identifier in the third task queue related to the second user, where the second user is one of the one or more other users.

[0011] In the above scheme, before obtaining the first inference result from the first task queue, the method further includes: using the second computing power resources subscribed by the second user to execute the inference task, obtaining the first inference result, and caching the first inference result in the first task queue.

[0012] This application provides a computer device, the computer device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the inference scheduling method provided in the embodiments of this application.

[0013] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the inference scheduling method provided in this application when executed by a processor.

[0014] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the inference scheduling method provided in this application.

[0015] This application provides an inference scheduling method, apparatus, computer device, computer-readable storage medium, and computer program product. Upon receiving an inference request from a first user, the method determines whether the available computing power resources of the first computing power resource subscribed to by the first user meet the computing power requirements of the inference request, thereby achieving dynamic management of the computing power resources of the cloud user. If the available computing power resources of the first computing power resource meet the computing power requirements of the inference request, the method updates the task queue set based on the inference request to achieve scheduling management of the cloud user's inference request. The method utilizes the first computing power resource to execute the inference task of the inference request, obtains the first inference result from the first task queue related to the inference result of the inference request, and feeds it back to the first user, achieving unified scheduling of cloud computing power resources and effectively solving the resource utilization problem in multi-user cloud scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the inference scheduling method according to an embodiment of this application; Figure 2 This is a statistical diagram of the first computing power resources in an embodiment of this application; Figure 3 This is a flowchart illustrating the execution of an inference task using the first computing resource in an embodiment of this application. Figure 4 This is a flowchart illustrating an example of an inference scheduling method applied according to an embodiment of this application; Figure 5 This is a schematic diagram of the composition structure of the inference scheduling device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Before providing a detailed description of the technical solution of this application, a brief introduction to the relevant technologies will be given first.

[0019] Currently, to improve the utilization rate of cloud computing resource pools, an online optimization scheduling method for AI inference clusters has been designed. This method involves configuring information on a pre-built inference cluster and a shared computing power resource pool, collecting the current status information of each GPU server in the inference cluster in real time, and allocating GPU servers that can currently provide inference services to the inference resource request information sent by external clients. During this process, the current load information of each GPU server is collected in real time. If the current load of the inference cluster reaches the upper limit threshold of its computing power capacity, a new GPU server is requested from the shared computing power resource pool. If the current load reaches the lower limit threshold of its computing power capacity, and there is a GPU server in the inference cluster that was requested from the shared computing power resource pool, a release operation is performed.

[0020] However, although this method can achieve resource sharing and allocation among different clusters, improve resource utilization, save actual project construction costs, and reduce resource management difficulty, management costs and energy consumption, it is only suitable for simple scenarios with a single cluster and a single tenant. It cannot effectively achieve unified scheduling in complex scenarios such as multi-tenant, multi-model, and cross-cluster environments in the cloud.

[0021] Based on this, this application proposes a reasoning scheduling method, which is applied to a cloud resource platform (hereinafter referred to as cloud platform) that can provide computing resources, such as a cloud vendor service platform or a cloud computing platform. Figure 1 This is a flowchart illustrating the inference scheduling method according to an embodiment of this application; as follows: Figure 1 As shown, the method includes: Step 101: Receive the inference request from the first user, determine whether the available computing resources in the first computing resources subscribed by the first user can meet the computing resources required for the inference request, and obtain the determination result; Step 102: If the judgment result indicates that the first computing power resource can meet the computing power resource required for the inference request, update the task queue set based on the inference request; wherein the task queue set includes at least a first task queue related to the inference result of the inference request; Step 103: Execute the reasoning task in the reasoning request using the first computing power resource to obtain the first reasoning result, which is cached in the first task queue; Step 104: Obtain the first inference result from the first task queue and send the first inference result to the first user.

[0022] In this embodiment, before executing the inference task using the first computing power resources subscribed by the first user after receiving the inference request, it is first determined whether the available computing power resources in the first computing power resources can meet the computing power resources required for the inference request, that is, whether the current available computing power resources of the first computing power resources are sufficient.

[0023] In this embodiment, the first user is any user or tenant who has subscribed to computing resources on the cloud platform. This application embodiment does not impose specific restrictions on the way users subscribe to computing resources. For example, users can purchase or rent corresponding GPU cards or NPU cards on the cloud platform according to their computing power type requirements to achieve the subscription of computing resources.

[0024] The first computing power resource corresponds to the first user, which refers to the computing power resource subscribed by the first user on the cloud platform. In this application embodiment, there is no specific limitation on the computing power resource subscribed by the user. For example, it can be the computing power resource corresponding to the high-performance GPU card or NPU card purchased by the user through the cloud platform.

[0025] Available computing resources in the first computing resource refer to the remaining available computing resources in the current first computing resource, that is, the available computing resources in the first computing resource excluding the computing resources currently occupied (or consumed). It should be noted that, in this embodiment of the application, in order to improve task processing efficiency, the cloud platform supports users to use the subscribed computing resources to process multiple inference tasks in parallel, so there will be computing resource occupancy.

[0026] The computing resources required for a reasoning request represent the estimated computing resources needed to execute the reasoning task in the reasoning request, that is, the computing resources expected to be needed to execute the reasoning task. The computing resources required for a reasoning request can be carried in the reasoning request.

[0027] It is understandable that before processing the inference task in a newly received inference request, it is necessary to determine whether the remaining computing power resources in the current first computing power resource meet the computing power requirements of the inference task. If the determination result is that the available computing power resources in the first computing power resource do not meet the computing power resources required by the inference request, it means that the remaining computing power resources in the current first computing power resource are limited and cannot provide computing power resources for the inference task in the inference request; if the determination result is that the available computing power resources in the first computing power resource meet the computing power resources required by the inference request, it means that the remaining computing power resources in the current first computing power resource are sufficient and can provide computing power resources for the inference task in the inference request.

[0028] It should be noted that the embodiments of this application do not impose specific restrictions on the method of receiving user inference requests. For example, a puller service can be created in the user cluster that has subscribed to computing resources, so that after a user submits an inference request through the platform or application programming interface (API), the puller can automatically pull the inference request to achieve dynamic management of multi-tenancy in the cloud.

[0029] In this embodiment, if it is determined from the judgment result that the first computing power resources can meet the computing power resources required for the inference request, a corresponding processing process or coroutine is created for the received inference request of the first user, and the task queue set is updated based on the inference request. The inference request is written into the task queue set for caching to complete the request enqueueing.

[0030] It should be noted that the embodiments of this application use asynchronous inference to process high-concurrency inference requests from cloud users, and create a set of task queues including multiple task queues, so as to cache various types of information related to each user's inference request through each task queue, thereby achieving unified management of high-concurrency inference requests.

[0031] In this embodiment, the task queue set includes at least a first task queue. The first task queue is related to the reasoning result of the reasoning request. That is, the first task queue is used to cache the reasoning result obtained by using computing resources to execute the reasoning task in the reasoning request. In other words, the first task queue is only used to store the reasoning result corresponding to the reasoning request.

[0032] It should be noted that this application embodiment does not impose specific restrictions on each task queue in the task queue set. For example, each task queue can adopt a key-value (KV) level caching method, such as a hash table caching structure. <key,queue <value>The specific process for updating the task queue set will be described in detail later in this application.

[0033] In this embodiment, after completing the process creation and queuing of the inference request, the inference task in the inference request is executed using the first computing power resource, and the obtained first inference result is cached in the first task queue for storing the inference result corresponding to the inference request.

[0034] It should be noted that the embodiments of this application do not impose specific restrictions on the process of the first computing power resource performing inference tasks. For example, after the first user subscribes to the first computing power resource on the cloud platform, a model service (Model-Server) and puller service corresponding to the first computing power resource can be created for the first user. The model service refers to the resource cluster corresponding to the first computing power resource, in which one or more models related to the computing power type requirements of the first user are deployed. When a request is enqueued, the puller service will automatically pull the inference task to be processed and push it to the model service. The model service will select the model to execute the inference task and send the inference result output by the model during the execution of the inference task back to the first task queue related to the inference request for caching in real time.

[0035] In this embodiment, after updating the task queue set based on the inference request, the processing process created for the inference request will wait for the inference result from the first task queue. During the execution of the inference task using the first computing power resources, the processing process will obtain the first inference result from the first task queue in real time and feed the first inference result back to the front-end interface of the first user until the inference task ends.

[0036] It should be noted that the inference scheduling method in this application embodiment can be implemented by an inference scheduling service module deployed on a cloud platform. The inference scheduling service module receives the inference request from the user, determines whether the available computing resources in the computing resources can meet the computing resources required for the inference request, and if it is determined that the computing resources can meet the computing resources required for the inference request, it updates the task queue set based on the inference request to complete the request enqueueing, uses computing resources to execute the inference task in the inference request, obtains the inference result from the task queue, and feeds the inference result back to the user's front-end interface until the inference task is completed.

[0037] In this embodiment, for a received inference request from a first user, the available computing resources of the first computing resource subscribed by the first user are determined to meet the computing resources required for the inference request, thereby achieving dynamic management of the computing resources of the cloud user. If the available computing resources of the first computing resource meet the computing power requirements of the inference request, the task queue set is updated based on the inference request to achieve scheduling management of the cloud user's inference request. The inference task of the inference request is executed using the first computing resource, and the first inference result is obtained from the first task queue related to the inference result of the inference request and fed back to the first user, thereby achieving unified scheduling of cloud computing resources and effectively solving the resource utilization problem in multi-user scenarios on the cloud.

[0038] In some optional implementations, the first computing power resource is represented as the number of tokens processed within the subscription period; before receiving the inference request from the first user, the method further includes: obtaining the total number of tokens corresponding to the first computing power resource within a preset period and the number of tokens consumed by the first computing power resource; and determining the number of available tokens corresponding to the first computing power resource within the preset period based on the total number of tokens and the number of tokens consumed.

[0039] In this embodiment, in order to realize the computing power statistics of multi-tenant scenarios in the cloud, the number of tokens is set as a unified measurement standard for computing power resources, so as to convert the computing power resources subscribed by different users into a unified token value, thereby realizing the cross-model computing power standardization of all types of GPU cards and NPU cards in the cloud infrastructure.

[0040] For example, the first computing power resource subscribed by the first user can be represented as the number of tokens that can be processed within the subscription period, that is, the number of tokens that the first user can use within a time unit within the subscription period. This application does not impose specific restrictions on the time unit, which can be determined by manual preset or based on the computing power demand of inference requests. For example, if the time unit is preset to minutes, the number of tokens processed per minute is used as the base computing power unit, or if the time unit is preset to seconds, the number of tokens processed per second is used as the base computing power unit. It should be noted that after the first user subscribes to the first computing power resource, the first computing power resource is converted into the total number of tokens (or total token computing power value) corresponding to the first computing power resource within the corresponding preset period, and the total number of tokens corresponding to the first computing power resource within the preset period is cached through the configuration file to ensure that the computing power quota of each user can be accurately obtained.

[0041] In this embodiment, before receiving the inference request from the first user, not only can the total number of tokens corresponding to the first computing power resource within a preset time period be obtained, but also the number of tokens consumed by the first computing power resource within a preset time period can be obtained. Based on the obtained total number of tokens and the number of tokens consumed, the number of available tokens corresponding to the first computing power resource within a preset time period can be determined for subsequent judgment on whether the computing power requirement of the inference request is met.

[0042] It should be noted that this application embodiment does not impose specific restrictions on the method of obtaining the number of consumption tokens of the first computing power resource. For example, the Prometheus monitoring tool can be deployed in the resource cluster corresponding to the first user's first computing power resource. Prometheus can be used to monitor and collect the resource consumption of the first user's first computing power resource when processing inference tasks, and convert it into the number of consumption tokens.

[0043] For example, the calculation process for the number of available tokens for the first computing power resource under a preset duration can be designed as follows: Available Tokens = Total Tokens - Tokens Consumed (1) The total number of tokens represents the total number of tokens corresponding to the first computing power resource within the preset duration, i.e., the total computing power input into the token pool within the preset duration; the consumed tokens represent the actual number of tokens consumed by the first computing power resource within the preset duration. By calculating the difference between the total number of tokens and the consumed tokens within the preset duration, the available tokens (or remaining tokens) of the first computing power resource within the preset duration can be obtained. It should be noted that in the process of calculating the available tokens of the first computing power resource, it is necessary to ensure that the preset duration of the total number of tokens and the consumed tokens are consistent. If the preset durations are inconsistent, a time unit consistency conversion should be performed, such as converting 5 minutes to 300 seconds, or 90 seconds to 1.5 minutes, etc.

[0044] It should be noted that after determining the number of available tokens for the first computing power resource within a preset time period based on the total number of tokens and the number of tokens consumed, the number of available tokens is pushed and cached to the inference scheduling service module. This allows the inference scheduling service module to determine whether the available computing power resources of the first computing power resource meet the computing power requirements of the inference request, thereby achieving real-time and accurate monitoring of the computing power resource usage of all users and ensuring efficient allocation and fair use of computing power resources in the future.

[0045] As an example, Figure 2 This is a statistical diagram of the first computing power resources in an embodiment of this application; as shown Figure 2 As shown, taking a preset duration of 5 minutes as an example, within 5 minutes, the total number of tokens for the first computing power resource subscribed by the first user is input into the token pool at fixed time intervals (i.e., 5 minutes). Prometheus monitors the number of tokens consumed within 5 minutes due to processing other inference tasks. When a new inference request is received (such as...), the token pool is updated accordingly. Figure 2 The arrow to the left of the token filter determines whether the number of available tokens for the first computing resource within 5 minutes meets the computing power requirements of the inference request. If the number of available tokens for the first computing resource is sufficient and meets the computing power requirements, then while continuing to execute other current inference tasks, the available computing power resources of the first computing resource are used to execute the inference task in the new inference request (such as...). Figure 2 (The arrow to the right of the token filter); if it is determined that the number of available tokens for the first computing power resource is limited and does not meet the computing power requirements, then the inference task in the new inference request will not be executed, and other current inference tasks will continue to be executed (such as...). Figure 2 (The arrow below the token filter).

[0046] It should be noted that, in this embodiment, the process of acquiring available computing resources for user-subscribed computing resources can be implemented by a computing power scheduling service (Qos-Service) module deployed on a cloud platform. This module focuses on computing power resource management in a multi-user environment on the cloud, integrating real-time computing power monitoring and computing power allocation functions. After a user subscribes to computing power resources, the computing power scheduling service module automatically converts the computing power resources into the total number of tokens corresponding to the computing power resources within a preset time period. It then caches the total number of tokens corresponding to the computing power resources within the preset time period through a configuration file and obtains the number of tokens consumed by the computing power resources within the preset time period through Prometheus. This allows the module to calculate the number of available tokens (i.e., available computing power resources) for the computing power resources within the preset time period and cache the number of available tokens to the inference scheduling service module. This allows the inference scheduling service module to determine whether the available computing power resources meet the computing power requirements of the inference request after receiving an inference request related to a user who has subscribed to the computing power resources. The computing power scheduling service module collects and analyzes the computing power resource usage or consumption of each user in real time, and dynamically adjusts the computing power allocation strategy to ensure efficient utilization and fair allocation of resources, providing stable and high-quality computing power support for inference tasks under multiple users.

[0047] In some optional implementations, the first computing power resource is represented as the number of tokens processed within the subscription period; determining whether the available computing power resources in the first computing power resource subscribed by the first user can meet the computing power resources required for the inference request includes: determining that the first computing power resource can meet the computing power resources required for the inference request when, within a preset period, the number of available tokens corresponding to the first computing power resource is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold; and determining that the first computing power resource cannot meet the computing power resources required for the inference request when, within the preset period, the number of available tokens corresponding to the first computing power resource is less than the first threshold, and / or the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is less than the second threshold.

[0048] In this embodiment, after obtaining the number of available tokens corresponding to the first computing power resource within the preset time period based on the total number of tokens and the number of tokens consumed by the first computing power resource within the preset time period, it is determined whether the computing power resource required for the inference request is met based on the number of available tokens and preset judgment conditions.

[0049] Specifically, the preset judgment conditions include one or more of the following: whether the number of available tokens corresponding to computing power resources within a preset time period is greater than or equal to a first threshold; whether the proportion of the number of available tokens corresponding to computing power resources within a preset time period to the total number of tokens is greater than or equal to a second threshold, wherein the first threshold represents the minimum number of available tokens, for example, it can be preset to 200; the second threshold represents the proportion of the minimum number of available tokens, for example, it can be preset to 5%. When all of the above judgment conditions are met, it is determined that the user's computing power resources can meet the computing power resources required for the inference request; when at least one of the above judgment conditions is not met, it is determined that the user's computing power resources cannot meet the computing power resources required for the inference request.

[0050] For example, taking the first computing power resource as an example, if the number of available tokens corresponding to the first computing power resource within a preset time period is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold, it is determined that the first computing power resource can meet the computing power resources required for the inference request; if the number of available tokens corresponding to the first computing power resource within a preset time period is less than the first threshold, and / or the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is less than the second threshold, it is determined that the first computing power resource cannot meet the computing power resources required for the inference request.

[0051] It should be noted that before determining whether the computing power resources required for the inference request are met based on the number of available tokens and preset judgment conditions, the number of available tokens of the first computing power resource within the preset time period can be compared with the computing power resources required for the inference request. The computing power resources required for the inference request can also be represented by the number of tokens within the preset time period. If the number of available tokens of the first computing power resource is greater than the number of tokens required for the inference request within the preset time period, then the subsequent judgment will be made based on the preset judgment conditions.

[0052] In some optional implementations, the inference request includes: a first user identifier corresponding to the first user, a first model identifier corresponding to the first computing resource, a first request identifier corresponding to the inference request, and inference request information including the inference task. The task queue set further includes a second task queue associated with the model and a third task queue associated with the first user. Updating the task queue set based on the inference request includes: constructing a first task queue associated with the first request identifier, the first task queue being used to cache the inference results obtained from executing the inference task; caching the first user identifier in the second task queue associated with the first model identifier; caching the first request identifier in the third task queue associated with the first user identifier; and generating a first mapping table based on the first request identifier and the inference request information.

[0053] In this embodiment, the received inference request from the first user specifically includes the user identifier corresponding to the first user, the first model identifier corresponding to the first computing power resource, the first request identifier corresponding to the inference request, and inference request information including the inference task.

[0054] The user identifier (account_ID) corresponding to the first user refers to the identity identifier of the first user, which is used to uniquely identify the identity of the first user.

[0055] The first model identifier (model_ID) corresponding to the first computing power resource refers to the resource cluster identifier of the first computing power resource subscribed by the first user. It is used to represent the model service (Model-Server) or resource cluster corresponding to the first computing power resource. It should be noted that in this embodiment of the application, users who subscribe to computing power resources of the same computing power type have the same model identifier.

[0056] The first request identifier (req_ID) corresponding to the inference request refers to the request identifier of the inference request currently sent by the first user, and corresponds to the first user.

[0057] The inference request message (req_message) refers to the specific information of the inference request, including the inference task, such as the text to be processed entered by the user through the front-end interface.

[0058] In this embodiment, if it is determined that the first computing power resource can meet the computing power resource required for the inference request, the information included in the inference request is written into the task queue set to update the task queue set and complete the enqueueing process of the inference request.

[0059] Specifically, a first task queue is constructed based on the first request identifier to cache the inference results obtained by the first computing power resource executing the inference task. The first user identifier is cached in the second task queue associated with the first model identifier, and the first request identifier is cached in the third task queue associated with the first user identifier. Based on the first request identifier and the inference request information, a first mapping table is constructed to cache the inference request information corresponding to the first request identifier.

[0060] In this embodiment, the first task queue associated with the first request identifier is used to cache the first inference result obtained by executing the inference task corresponding to the first request identifier using the first computing power resources; that is, it is used to record all inference results corresponding to the first request identifier. For example, taking the first task queue as an example where a hash table is used for caching, the first task queue can be represented as follows: <req_ID_resp,queue <response>>, where the key value req_ID_resp of the first task queue represents the response information corresponding to the first request identifier (req_ID), and the value value response of the first task queue represents the response token, that is, the specific first inference result.

[0061] The second task queue, associated with the first model identifier, is used to cache the user identifiers corresponding to the first model identifier; that is, it records all users who have subscribed to computing power resources of the same computing power type. For example, taking a hash table as an example, the second task queue can be represented as follows: <model_ID,queue<account_ID> During the request enqueueing process, the user identifier (account_ID) of the corresponding first user is written into the second task queue according to the key value model_ID of the second task queue.

[0062] A third task queue associated with the first user identifier is used to cache inference requests corresponding to the first user identifier, that is, to record all inference requests from the first user. For example, taking a hash table as an example, the third task queue can be represented as follows: <account_ID,queue<req_ID> During the request enqueueing process, the corresponding first request identifier (req_ID) is written into the third task queue according to the key value account_ID of the third task queue.

[0063] The first mapping table is associated with the first request identifier and inference request information. It is used to cache the inference request information corresponding to the first request identifier, that is, to record the complete information of the inference request corresponding to the first request identifier, to ensure the accuracy of the inference request information. For example, the first mapping table can be...<req_ID,req_message> The cache is in the form of key-value pairs. During the request enqueueing process, the first request identifier (req_ID) and the inferred request information (req_message) are generated in the form of key-value pairs to form the first mapping table.

[0064] It should be noted that the task queue set in this embodiment can be stored in the cache layer of the cloud platform, and the unified management of each task queue in the task queue set can be realized in the cache layer to improve the efficiency of request enqueuing.

[0065] Before using the first computing power resource to execute the inference task in the inference request, the method further includes: determining whether the number of tasks being processed by the first computing power resource has reached a preset maximum number of tasks; if it is determined that the preset maximum number of tasks has not been reached, obtaining the first request identifier from the third task queue based on the first user identifier, and obtaining the inference task from the first mapping table based on the first request identifier.

[0066] In this embodiment, before pulling inference tasks through the puller to schedule the first computing power resource for processing, it is necessary to determine whether the number of inference tasks currently existing in the first computing power resource has reached the processing limit. Specifically, this is done by monitoring the number of tasks currently being processed by the first computing power resource and comparing it with the preset maximum number of tasks. If the number of tasks being processed has not reached the preset maximum number of tasks, the puller actively pulls inference tasks from the updated task queue set and calls the first computing power resource to execute the inference task. If the number of tasks being processed has reached the preset maximum number of tasks, the puller pauses pulling inference tasks, waits for an available processing slot to appear, and then pulls inference tasks again and performs subsequent task processing.

[0067] Specifically, the process of the puller pulling inference tasks from the updated task queue set is as follows: based on the first user identifier, the first request identifier corresponding to the inference request is read from the third task queue; based on the first request identifier, the inference request information is read from the first mapping table to obtain the inference task in the inference request; and then the read inference task is pushed as input information to the resource cluster corresponding to the first computing power resource for model inference.

[0068] As an example, Figure 3 This is a flowchart illustrating the execution of an inference task using the first computing resource in an embodiment of this application; as shown below. Figure 3 As shown, when the first computing power resources subscribed by the first user can meet the computing power resources required for the inference request, the specific process of executing the inference task through the inference scheduling service module is as follows: Step 201: Obtain the inference request.

[0069] Specifically, the inference scheduling service module receives an inference request sent by the first user through an API interface. After obtaining the number of available tokens corresponding to the first computing power resource within a preset time period, it determines whether the required computing power resources for the inference request are met based on the number of available tokens and preset judgment conditions. If the number of available tokens corresponding to the first computing power resource within the preset time period is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold, then it is determined that the first computing power resource can meet the required computing power resources for the inference request. (The process of determining whether the available computing power resources in the first computing power resource meet the required computing power resources for the inference request is not included in the...) Figure 3 (as reflected in the text).

[0070] Step 202: Create the first task queue.

[0071] Specifically, if it is determined that the first computing power resources subscribed by the first user can meet the computing power resources required for the inference request, a processing process associated with the first user is created, and the task queue set stored in the cache layer is updated based on the inference request. First, a first task queue associated with the first request identifier is constructed based on the first request identifier included in the inference request, so as to cache the inference results obtained by the first computing power resources executing the inference task.

[0072] Step 203: Cache the first user identifier.

[0073] Specifically, the first user identifier corresponding to the first user included in the inference request is cached in the second task queue associated with the first model identifier. That is, according to the key value model_ID of the second task queue, the user identifier (account_ID) of the corresponding first user is written into the second task queue.

[0074] Step 204: Cache the first request identifier.

[0075] Specifically, the first request identifier corresponding to the inference request included in the inference request is cached in the third task queue associated with the first user identifier, and the corresponding first request identifier (req_ID) is written into the third task queue according to the key value account_ID of the third task queue.

[0076] Step 205: Generate the first mapping table.

[0077] Specifically, based on the first request identifier and inference request information included in the inference request, a first mapping table is constructed to cache the complete inference request information corresponding to the first request identifier. That is, the first request identifier (req_ID) and the inference request information (req_message) are generated in the form of key-value pairs to form the first mapping table.

[0078] Step 206: Read the first request identifier.

[0079] Specifically, after updating the task queue set based on the inference request, the puller deployed in the resource cluster 1 created for the first computing power resource will automatically pull the inference tasks from the inference request information if it determines that the number of existing inference tasks of the first computing power resource has not reached the processing limit. In the process of pulling the inference tasks, the first request identifier cached in the third task queue is read based on the first user identifier.

[0080] Step 207: Retrieve the reasoning task.

[0081] Specifically, after reading the first request identifier cached in the third task queue, the puller automatically pulls the inference task from the inference request information from the inference scheduling service module by reading the inference request information in the first mapping table based on the first request identifier.

[0082] Furthermore, after the puller retrieves the inference task through the inference scheduling service module, it pushes the inference task to the model service corresponding to the first computing power resource subscribed by the first user, so as to perform model inference on the inference task and obtain the first inference result.

[0083] Step 208: Report the first reasoning result.

[0084] Specifically, during the process of performing model inference on the inference task through the model service corresponding to the first computing power resource, the first inference result corresponding to the inference request is sent back to the inference scheduling service module in real time.

[0085] Step 209: Cache the first inference result.

[0086] Specifically, after obtaining the first inference result corresponding to the inference request, the inference scheduling service module caches the first inference result in the first task queue associated with the first request identifier.

[0087] Step 210: Return the first reasoning result.

[0088] Specifically, after caching the first inference result to the first task queue associated with the first request identifier, the processing process retrieves the inference result from the task queue and feeds the first inference result back to the front-end interface of the first user until the inference task is completed.

[0089] In some optional implementations, the task queue set further includes a second task queue related to the model and a third task queue related to the user, and the inference request includes a first request identifier corresponding to the inference request; when the judgment result indicates that the first computing power resources cannot meet the computing power resources required by the inference request, the method further includes: determining one or more other user identifiers other than the first user identifier corresponding to the first user based on the second task queue, and judging whether the available computing power resources in the second computing power resources subscribed by each other user can meet the computing power resources required by the inference request based on each other user identifier; when the available computing power resources in the second computing power resources subscribed by the second user can meet the computing power resources required by the inference request, caching the first request identifier in the third task queue related to the second user, where the second user is one of the one or more other users.

[0090] In this embodiment, if it is determined from the judgment result that the first computing power resource cannot meet the computing power resource required for the inference request, the first computing power resource cannot execute the inference task in the first user's inference request. The second computing power resource of the second user that can meet the first user's inference request will be selected according to the task queue set, so as to borrow the second computing power resource subscribed by the second user to execute the inference task in the first user's inference request, thereby realizing the dynamic scheduling of cross-user computing power.

[0091] Specifically, based on the first model identifier, one or more other user identifiers besides the first user identifier are searched from all user identifiers stored in the second task queue. That is, other users who have subscribed to computing power resources of the same computing power type as the first user are identified from the second task queue. It is then determined whether the available computing power resources in the second computing power resources subscribed by each other user meet the computing power resources required for the first user's inference request, thereby identifying the second user who meets the conditions. The task queue cluster is updated according to the user identifier of the second user so as to use the second computing power resources subscribed by the second user to execute the inference task requested by the first user.

[0092] It should be noted that, in this embodiment of the application, the process of determining whether the available computing resources in the second computing resources subscribed by each other user meet the computing resources required for the inference request of the first user is the same as the process of determining whether the available computing resources in the first computing resources meet the computing resources required for the inference request. The computing power scheduling service module obtains the number of available tokens in the second computing resources for each other user within a preset time period, and makes a judgment based on the number of available tokens and preset conditions. The preset conditions have been described in detail above and will not be repeated here.

[0093] In this embodiment, after determining the second user who meets the required computing power resources for the inference request from one or more other users through the above judgment process, the specific process of updating the task queue cluster according to the user identifier of the second user is as follows: the first request identifier corresponding to the inference request from the first user is cached in the third task queue related to the second user included in the task queue set, thereby writing the inference request of the first user into the task queue related to the second user.

[0094] The third task queue associated with the second user is used to cache inference requests corresponding to the second user's identifier. That is, it is used to record all inference requests from the second user. When the second user's second computing power resources are used to process the inference task of the first user's inference request, the first request identifier (req_ID) corresponding to the first user's inference request will be written into the third task queue associated with the second user according to the key value account_ID of the third task queue.

[0095] In some optional implementations, before obtaining the first inference result from the first task queue, the method further includes: using the second computing power resources subscribed by the second user to execute the inference task, obtaining the first inference result, and caching the first inference result in the first task queue.

[0096] In this embodiment, when it is determined that the second computing power resources subscribed by the second user are needed to meet the computing power requirements of the first user's inference request, the first user's inference request is written into the task queue related to the second user. Then, the inference task is executed using the second computing power resources subscribed by the second user to obtain the first inference result. The first inference result is cached in the first task queue associated with the first request identifier. The processing process created for the inference request will wait for the inference result from the first task queue. During the execution of the inference task using the second computing power resources, the processing process obtains the first inference result from the first task queue in real time and feeds the first inference result back to the front-end interface of the first user until the inference task ends, so as to realize the dynamic scheduling of cross-user computing power.

[0097] It should be noted that the embodiments of this application do not impose specific restrictions on the process of the second computing power resource executing the inference task from the inference request of the first user. It can be the same as the process of the first computing power resource executing the inference task. After the second user subscribes to the second computing power resource on the cloud platform, the cloud platform can create a model service and a puller service corresponding to the second computing power resource. When a request is enqueued, the puller service will automatically pull the inference task to be processed and push it to the model service. The model service will select the model to execute the inference task and send the inference result output by the model during the execution of the inference task back to the first task queue related to the inference request for caching in real time.

[0098] In this embodiment, after updating the task queue set based on the user identifier of the second user, the specific process of the puller deployed in the model service of the second computing power resource pulling the inference task of the first user from the updated task queue set is as follows: based on the user identifier of the second user, the first request identifier corresponding to the inference request of the first user is read from the third task queue; based on the first request identifier, the inference request information is read from the first mapping table to obtain the inference task in the inference request; and then the read inference task is pushed as input information to the resource cluster corresponding to the second computing power resource for model inference.

[0099] It should be noted that the dynamic scheduling method for cross-user computing power in this embodiment can be implemented by an inference scheduling service module deployed on a cloud platform. When it is determined that the computing power resources subscribed by a user cannot meet the computing power resources required for the inference request, the inference scheduling service module determines other users who have subscribed to the same type of computing power resources in the second task queue associated with the model identifier corresponding to the user. By judging the computing power resources subscribed by other users, the computing power resources of the second user that meet the computing power requirements of the user's inference request are determined. The user's inference request is written into the task queue related to the second user, and the computing power resources of the second user are used to execute the inference task. The inference result is obtained from the task queue and fed back to the user's front-end interface until the inference task ends.

[0100] In this embodiment of the application, when the computing power resources currently subscribed by a user cannot meet the computing power resources required for the inference request sent by the user, computing power borrowing is used to realize the dynamic scheduling of computing power across users, which effectively improves the utilization rate of computing power resources of the cloud platform and enhances the overall performance and user experience of model inference services in the cloud environment.

[0101] As an example, Figure 4 This is a flowchart illustrating an example of an inference scheduling method applied to an embodiment of this application; as shown below. Figure 4 As shown, the specific process of implementing inference scheduling through the computing power scheduling service module and the inference scheduling service module is as follows: Step 301: After the first user subscribes to the first computing power resource, determine the total number of tokens within the preset time period.

[0102] Specifically, after the first user subscribes to the first computing power resource, the computing power scheduling service module converts the first computing power resource into the total number of tokens corresponding to the first computing power resource within a corresponding preset time period, and caches the total number of tokens corresponding to the first computing power resource within the preset time period through the configuration file.

[0103] Step 302: Obtain the number of tokens consumed by the first computing power resource within the preset time period, and determine the number of available tokens.

[0104] Specifically, the computing power scheduling service module monitors and collects the resource consumption of the first user's first computing power resource when processing inference tasks through Prometheus, obtains the number of tokens consumed by the first computing power resource within a preset time period, and determines the number of available tokens for the first computing power resource within the preset time period based on the total number of tokens and the number of tokens consumed by the first computing power resource within the preset time period.

[0105] Step 303: Send the number of available tokens for the first computing power resource within a preset time period to the inference scheduling service module.

[0106] Specifically, after obtaining the number of available tokens for the first computing power resource within a preset time period, the computing power scheduling service module sends the number of available tokens for the first computing power resource within a preset time period to the inference scheduling service module so as to cache the number of available tokens in the inference scheduling service module.

[0107] Step 304: Obtain the inference request from the first user.

[0108] Specifically, the inference scheduling service module receives an inference request sent by the first user. The inference request may include the first user identifier and the first model identifier corresponding to the first user, as well as the first request identifier, inference request information, and required computing resources corresponding to the inference request.

[0109] Step 305: Determine whether the available computing resources of the first computing resource meet the computing resources required for the inference request.

[0110] Specifically, the inference scheduling service module queries the number of available tokens for the first computing power resource in the cache within a preset time period, and determines whether the computing power resources required for the inference request are met based on the number of available tokens and preset judgment conditions. For the specific judgment process, please refer to step 201, which will not be elaborated here.

[0111] If the judgment result determines that the first computing power resources subscribed by the first user can meet the computing power resources required for the inference request, proceed to step 306; if the judgment result determines that the first computing power resources subscribed by the first user cannot meet the computing power resources required for the inference request, proceed to step 307.

[0112] Step 306: Update the task queue set based on the inference request.

[0113] Specifically, if it is determined that the first computing power resources subscribed by the first user can meet the computing power resources required for the inference request, a processing process associated with the first user is created, and the task queue set stored in the cache layer is updated based on the inference request. For the specific processing process, refer to steps 202 to 205, which will not be elaborated here.

[0114] Step 307: Select a second user and update the task queue set based on the second user and the inference request.

[0115] Specifically, if it is determined that the first computing power resources subscribed by the first user cannot meet the computing power resources required for the inference request, based on the first model identifier, other users who have subscribed to computing power resources of the same type as the first user are identified from all user identifiers stored in the second task queue. It is then determined whether the available computing power resources in the second computing power resources subscribed by each other user meet the computing power resources required for the first user's inference request, thereby identifying the second user who meets the conditions. The first request identifier corresponding to the inference request from the first user is cached in the third task queue related to the second user included in the task queue set, thereby writing the first user's inference request into the task queue related to the second user.

[0116] Step 308: Read the reasoning task.

[0117] Specifically, when a puller in the resource cluster corresponding to the first computing resource subscribed by the first user or the second computing resource subscribed by the second user requests to pull an inference task, the inference scheduling service module reads the first request identifier cached in the third task queue associated with the first or second user based on the first user identifier or the second user identifier. Based on the first request identifier, it reads the inference task from the inference request information in the first mapping table, so that the puller can automatically pull the inference task from the inference scheduling service module. The specific processing steps are detailed in steps 206 to 207, and will not be elaborated further here.

[0118] Step 309: Receive the first inference result and cache it in the first task queue.

[0119] Specifically, after the puller in the resource cluster corresponding to the first computing resource subscribed by the first user or the second computing resource subscribed by the second user pulls the inference task, during the model inference process of the inference task through the model service corresponding to the first computing resource or the second computing resource, the first inference result corresponding to the inference request is sent back to the inference scheduling service module in real time. The inference scheduling service module caches the received first inference result in the first task queue associated with the first request identifier. For details, refer to steps 208 to 209, which will not be elaborated here.

[0120] Step 310: Retrieve the first inference result from the first task queue and return it to the first user.

[0121] Specifically, after the inference scheduling service module caches the first inference result obtained into the first task queue associated with the first request identifier, the processing process retrieves the inference result from the task queue and feeds the first inference result back to the front-end interface of the first user until the inference task ends.

[0122] In this embodiment, a dual-coroutine collaboration mechanism is designed between the computing power scheduling service module and the inference scheduling service module. The computing power scheduling service module converts computing power into tokens to achieve computing power statistics and scheduling for multi-user scenarios in the cloud. The inference scheduling service module builds a unified computing power resource scheduling layer to achieve unified scheduling of computing power resources in the cloud, supports dynamic management of computing power resources for multiple users in the cloud, ensures high service availability and flexible adjustment of computing resources, effectively decouples the relationship between the basic resource layer and the scheduling logic layer, realizes unified integration and scheduling of computing power resources of the cloud platform, and improves the utilization rate and flexibility of computing power resources of cloud infrastructure.

[0123] Based on the above embodiments, this application also provides an inference scheduling device. Figure 5 This is a schematic diagram of the composition structure of the inference scheduling device provided in the embodiments of this application; as shown below. Figure 5 As shown, the device includes: a communication unit 41, a scheduling unit 42, and an inference unit 43; wherein, The communication unit 41 is configured to receive a reasoning request from a first user; and to obtain a first reasoning result from a first task queue and send the first reasoning result to the first user. The scheduling unit 42 is configured to determine whether the available computing resources in the first computing resources subscribed by the first user can meet the computing resources required for the inference request, and obtain a determination result; it is also configured to update the task queue set based on the inference request when the determination result indicates that the first computing resources can meet the computing resources required for the inference request; wherein the task queue set includes at least a first task queue related to the inference result of the inference request. The inference unit 43 is used to execute the inference task in the inference request using the first computing power resource to obtain a first inference result, and the first inference result is cached in the first task queue.

[0124] In one optional embodiment of this application, the first computing power resource is represented as the number of tokens processed within the subscription period; the communication unit 41 is further configured to obtain the total number of tokens corresponding to the first computing power resource within a preset period and the number of tokens consumed by the first computing power resource before receiving the inference request from the first user; The scheduling unit 42 is further configured to determine the number of available tokens corresponding to the first computing power resource within the preset time period based on the total number of tokens and the number of tokens consumed.

[0125] In one optional embodiment of this application, the first computing power resource is represented as the number of tokens processed within the subscription period; the scheduling unit 42 is configured to determine that the first computing power resource can meet the computing power resources required for the inference request when, within a preset period, the number of available tokens corresponding to the first computing power resource is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold; and to determine that the first computing power resource cannot meet the computing power resources required for the inference request when, within the preset period, the number of available tokens corresponding to the first computing power resource is less than the first threshold, and / or the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is less than the second threshold.

[0126] In one optional embodiment of this application, the inference request includes: a first user identifier corresponding to the first user, a first model identifier corresponding to the first computing resource, a first request identifier corresponding to the inference request, and inference request information including the inference task. The task queue set further includes a second task queue associated with the model and a third task queue associated with the first user. The scheduling unit 42 is used to construct a first task queue associated with the first request identifier, the first task queue being used to cache the inference results obtained from executing the inference task; cache the first user identifier in the second task queue associated with the first model identifier; cache the first request identifier in the third task queue associated with the first user identifier; and generate a first mapping table based on the first request identifier and the inference request information.

[0127] In an optional embodiment of this application, the scheduling unit 42 is further configured to determine whether the number of tasks being processed by the first computing power resource has reached a preset maximum number of tasks before executing the inference task in the inference request using the first computing power resource; if it is determined that the preset maximum number of tasks has not been reached, obtain the first request identifier from the third task queue based on the first user identifier, and obtain the inference task from the first mapping table based on the first request identifier.

[0128] In an optional embodiment of this application, the task queue set further includes a second task queue related to the model and a third task queue related to the user. The inference request includes a first request identifier corresponding to the inference request. The scheduling unit 42 is further configured to, when the judgment result indicates that the first computing power resources cannot meet the computing power resources required for the inference request, determine one or more other user identifiers other than the first user identifier corresponding to the first user based on the second task queue, and determine whether the available computing power resources in the second computing power resources subscribed by each other user can meet the computing power resources required for the inference request based on each other user identifier; when the available computing power resources in the second computing power resources subscribed by the second user can meet the computing power resources required for the inference request, cache the first request identifier in the third task queue related to the second user, wherein the second user is one of the one or more other users.

[0129] In an optional embodiment of this application, the inference unit 43 is further configured to execute the inference task using the second computing power resources subscribed by the second user before obtaining the first inference result from the first task queue, and obtain the first inference result, wherein the first inference result is cached in the first task queue.

[0130] In this application, the scheduling unit 42 and inference unit 43 in the device can be implemented by a CPU, digital signal processor (DSP), microcontroller unit (MCU), or field-programmable gate array (FPGA) in practical applications; the communication unit 41 in the device can be implemented by a communication module (including: basic communication kit, operating system, communication module, standardized interface and protocol, etc.) and transceiver antenna in practical applications.

[0131] It should be noted that the above-described inference scheduling device is only illustrated by the division of the program modules. In practical applications, the processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the inference scheduling device and the inference scheduling method embodiments provided above belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0132] This application also provides a computer device. Figure 6 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application; as shown below. Figure 6 As shown, the computer device includes a memory 52, a processor 51, and a computer program stored in the memory 52 and executable on the processor 51.

[0133] Optionally, the computer device also includes at least one communication component 54. The various components in the computer device can be coupled together via a bus system 53. It is understood that the bus system 53 is used to implement communication between these components. In addition to a data bus, the bus system 53 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general labeled all buses as Bus System 53.

[0134] It is understood that memory 52 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 52 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0135] The methods disclosed in the embodiments of this application can be applied to processor 51, or implemented by processor 51. Processor 51 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 51 or by instructions in the form of software. The processor 51 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 51 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 52. ​​Processor 51 reads the information in memory 52 and combines its hardware to complete the steps of the aforementioned method.

[0136] In an exemplary embodiment, the computer device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned methods.

[0137] This application also provides a computer-readable storage medium on which a computer program is stored.

[0138] Optionally, the computer-readable storage medium can be applied to the inference scheduling apparatus of the present application embodiment; then when the program is executed by the processor, it implements the steps of the inference scheduling method of the present application embodiment.

[0139] This application also provides a computer program product, including a computer program that can be executed by the processor 51 of a computer device to complete the steps of the inference scheduling method described in this application.

[0140] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0141] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0142] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0144] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0146] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0148] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.< / response> < / value>

Claims

1. A reasoning scheduling method, characterized in that, The method includes: Receive the inference request from the first user, determine whether the available computing resources in the first computing resources subscribed by the first user can meet the computing resources required by the inference request, and obtain the determination result; If the determination result indicates that the first computing power resource can meet the computing power resource required for the inference request, the task queue set is updated based on the inference request; wherein, the task queue set includes at least the first task queue related to the inference result of the inference request; The first computing power resource is used to execute the reasoning task in the reasoning request to obtain a first reasoning result, which is cached in the first task queue. The first inference result is obtained from the first task queue and sent to the first user.

2. The method according to claim 1, characterized in that, The first computing power resource is represented by the number of tokens processed within the subscription period; before receiving the inference request from the first user, the method further includes: Obtain the total number of tokens corresponding to the first computing power resource and the number of tokens consumed by the first computing power resource within a preset time period; Based on the total number of tokens and the number of tokens consumed, the number of available tokens corresponding to the first computing power resource within the preset time period is determined.

3. The method according to claim 1 or 2, characterized in that, The first computing power resource is represented by the number of tokens processed within the subscription period; determining whether the available computing power resources in the first computing power resource subscribed by the first user can meet the computing power resources required for the inference request includes: If, within a preset time period, the number of available tokens corresponding to the first computing power resource is greater than or equal to a first threshold, and the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is greater than or equal to a second threshold, it is determined that the first computing power resource can meet the computing power resources required for the inference request. If, within the preset time period, the number of available tokens corresponding to the first computing power resource is less than a first threshold, and / or the proportion of the number of available tokens corresponding to the first computing power resource to the total number of tokens is less than a second threshold, it is determined that the first computing power resource cannot meet the computing power resources required for the inference request.

4. The method according to claim 1, characterized in that, The inference request includes: a first user identifier corresponding to the first user, a first model identifier corresponding to the first computing resource, a first request identifier corresponding to the inference request, and inference request information including the inference task. The task queue set also includes a second task queue related to the model and a third task queue related to the first user. Updating the task queue set based on the inference request includes: Construct a first task queue associated with the first request identifier, the first task queue being used to cache the inference results obtained from executing the inference task; Cache the first user identifier in the second task queue associated with the first model identifier; The first request identifier is cached in the third task queue associated with the first user identifier; A first mapping table is generated based on the first request identifier and the inference request information.

5. The method according to claim 4, characterized in that, Before utilizing the first computing power resource to execute the inference task in the inference request, the method further includes: Determine whether the number of tasks being processed by the first computing power resource has reached the preset maximum number of tasks. If it is determined that the preset maximum number of tasks has not been reached, obtain the first request identifier from the third task queue based on the first user identifier, and obtain the inference task from the first mapping table based on the first request identifier.

6. The method according to claim 1, characterized in that, The task queue set also includes a second task queue related to the model and a third task queue related to the user, and the inference request includes a first request identifier corresponding to the inference request; If the determination result indicates that the first computing power resource cannot meet the computing power resource required for the inference request, the method further includes: Based on the second task queue, determine one or more other user identifiers besides the first user identifier corresponding to the first user, and based on each other user identifier, determine whether the available computing resources in the second computing resources subscribed by each other user can meet the computing resources required for the inference request. When the available computing resources in the second computing resources subscribed by the second user are sufficient to meet the computing resources required for the inference request, the first request identifier is cached in the third task queue related to the second user, where the second user is one of one or more other users.

7. The method according to claim 6, characterized in that, Before obtaining the first inference result from the first task queue, the method further includes: The reasoning task is executed using the second computing resources subscribed by the second user to obtain a first reasoning result, which is then cached in the first task queue.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images