Scheduling method and device for large language model reasoning system with multiple processing units

Abstract

The invention relates to a scheduling method and device for a large language model reasoning system with multiple processing units, and the method comprises the steps: obtaining a reasoning request of a user, the reasoning request comprising a text input by the user; classifying the reasoning requests according to the content of the text by utilizing a classification model so as to determine a request category to which the reasoning requests belong in at least one preset request category; the scheduling score of each processing unit for the request category is obtained, and the scheduling score of each processing unit for the request category is based on the satisfaction score of the processing unit for the request category; the inference request is scheduled to one of the plurality of processing units based on the scheduling score for the request category for each of the plurality of processing units. According to the method and the device, the reasoning requests can be dynamically scheduled to different types of processing units according to the request types to which the reasoning requests belong, so that reasoning services with higher satisfaction and cost performance are provided for users.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a scheduling method and device for a large language model inference system with multiple processing units. Background Art

[0002] Large language models (LLMs) have been widely used in various application scenarios due to their powerful natural language processing capabilities. Reasoning with large language models requires a large number of processing units, such as graphics processing units (GPUs). Traditionally, large language models have been deployed on a single type of GPU for reasoning, ignoring the optimization potential offered by hardware diversity. With the development of cloud computing and data centers, chip heterogeneity—that is, deploying large language models on multiple GPUs for reasoning—has become increasingly common.

[0003] In the related art, in a large language model reasoning system including multiple processing units, usually only the reasoning effect is considered and the user's reasoning requests are more often dispatched to the processing unit with better reasoning effect. Although this can ensure the quality of the reasoning results, it may cause the reasoning cost to be too high. Alternatively, only the reasoning cost is considered and more reasoning requests are dispatched to the processing unit with lower reasoning cost. Although this can reduce the reasoning cost to a certain extent, it may cause the quality of the reasoning result to be low. Alternatively, the reasoning request is randomly dispatched to any processing unit without considering the reasoning effect and reasoning cost. This may be able to control the reasoning effect and reasoning cost overall, but it cannot fully utilize the performance or cost advantages of each processing unit. In general, these scheduling methods do not consider user satisfaction to provide users with more cost-effective reasoning services. Summary of the Invention

[0004] In view of this, the present disclosure proposes a scheduling method, electronic device and storage medium for a large language model inference system with multiple processing units, which can dynamically schedule inference requests to different types of processing units according to the request category to which the inference requests belong, in order to provide users with more satisfactory and cost-effective inference services.

[0005] According to one aspect of the present disclosure, a scheduling method for a large language model inference system having multiple processing units is provided, wherein different types of processing units in the multiple processing units have different hardware architectures, and a trained large language model is deployed on each of the multiple processing units. The method includes:

[0006] Obtaining a user's reasoning request, where the reasoning request includes text input by the user;

[0007] Using a classification model, classifying the inference request according to the content of the text, so as to determine the request category to which the inference request belongs in at least one preset request category;

[0008] Obtaining a scheduling score for each of the multiple processing units for the request category, wherein the scheduling score for each processing unit for the request category is based on a satisfaction score for the request category obtained by the processing unit, the satisfaction score being calculated based on user ratings of processing results obtained by the processing unit when previously processing inference requests for the request category;

[0009] The inference request is dispatched to one of the multiple processing units based on a dispatch score of each of the multiple processing units for the request category, so that the large language model deployed by the one processing unit processes the inference request.

[0010] In one possible implementation, scheduling the inference request to one of the multiple processing units based on a scheduling score of each of the multiple processing units for the request category includes:

[0011] The ratio of the scheduling score of each processing unit for the request category to the total scheduling scores of the multiple processing units for the request category is used as the scheduling probability of the processing unit for the request category, and the inference request is scheduled to one of the multiple processing units according to the scheduling probability of each processing unit among the multiple processing units for the request category.

[0012] In a possible implementation, the scheduling score of each processing unit for the request category is equal to the satisfaction score of the processing unit for the request category.

[0013] In one possible implementation, the scheduling score of each processing unit for the request category is calculated by multiplying the satisfaction score of the processing unit for the request category and the resource utilization rate of the processing unit by a preset weight, and then summing the results to obtain a weighted sum of the satisfaction score and the resource utilization rate.

[0014] The resource utilization rate is calculated based on the number of used processing units of the type of processing units and the total number of the type of processing units.

[0015] In one possible implementation, the satisfaction score of each of the multiple processing units for the request category is the average of user ratings of one or more processing results obtained by the processing unit in processing one or more inference requests of the request category.

[0016] In a possible implementation, the method further includes:

[0017] After the large language model deployed in the one processing unit processes the inference request to obtain a processing result, obtaining a user rating of the processing result by a user;

[0018] The user score is stored in association with the type of the processing unit and the request category to which the inference request belongs.

[0019] In a possible implementation, the method further includes:

[0020] At regular time intervals, updating the satisfaction score of each processing unit in the plurality of processing units for each request category in the at least one request category, the updating comprising:

[0021] Obtaining user scores of one or more processing results obtained by each of the multiple processing units processing one or more inference requests of each request category in the at least one request category within the fixed time period;

[0022] For each request category, the satisfaction score of each processing unit for the request category is updated to the average user rating of one or more processing results obtained by each processing unit processing one or more inference requests of the request category within the fixed time period.

[0023] In a possible implementation, the method further includes:

[0024] analyzing a change trend of the satisfaction score according to the satisfaction score of each processing unit of the multiple processing units for each request category of the at least one request category within a plurality of consecutive fixed time periods;

[0025] When the scheduling score of each processing unit for the request category is the weighted sum of the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit, based on the changing trend of the satisfaction score, adjust the preset weights of the satisfaction score and the resource utilization when calculating the weighted sum.

[0026] According to another aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0027] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0028] Through the above-mentioned aspects of the present disclosure, it is possible to determine the request category to which the user's reasoning request belongs, and based on the scheduling score of each processing unit for the request category, dynamically schedule the user's reasoning request to one of a plurality of processing units, so as to avoid randomly scheduling the user's reasoning request to any processing unit without considering user satisfaction, or scheduling too many reasoning requests to processing units with better reasoning effects, resulting in excessively high reasoning costs, or scheduling too many reasoning requests to processing units with lower reasoning costs and ignoring the reasoning effects, thereby providing users with reasoning services with higher satisfaction and cost-effectiveness.

[0029] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0031] Figure 1 A schematic diagram of the structure of a large language model inference system according to an embodiment of the present disclosure is shown;

[0032] Figure 2 A flowchart showing a scheduling method for a large language model inference system with multiple processing units according to an embodiment of the present disclosure is shown;

[0033] Figure 3 A flowchart illustrating another scheduling method for a large language model inference system with multiple processing units according to an embodiment of the present disclosure is shown;

[0034] Figure 4 A structural diagram of a scheduling device for a large language model inference system with multiple processing units according to an embodiment of the present disclosure is shown;

[0035] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0037] As used herein, the terms "comprises," "comprising," "having," or variations thereof are open ended and include one or more stated features, integers, elements, steps, parts, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, parts, functions, or groups thereof.

[0038] When an element is referred to as being "connected," "coupled," "responsive" or variations thereof to another element, it can be directly connected, coupled or responsive to the other element or intervening elements may be present.

[0039] Although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teachings of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments.

[0040] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0042] In the present disclosure, a large language model inference system can have multiple processing units, different types of processing units within the multiple processing units have different hardware architectures, and each of the multiple processing units is deployed with a trained large language model. The processing unit can be, for example, a GPU, or other processing unit capable of performing large language model inference, such as an LPU or CPU. For ease of understanding, the following description uses a GPU as an example.

[0043] Figure 1 FIG. 1 shows a schematic diagram of a large language model inference system according to an embodiment of the present disclosure. Figure 1As shown, the large language model inference system can include multiple GPUs (i.e., GPU-A, GPU-B, GPU-C, etc. in the figure) as multiple processing units. Different types of processing units have different hardware architectures, which also causes different types of processing units to have different performance, power consumption, and costs when processing the same inference request. For example, different types of processing units can be distinguished by brand and model.

[0044] Each of the multiple processing units in the large language model inference system may include a certain number of such processing units. Figure 1 Among them, GPU-A type GPUs can include GPU-A1, GPU-A2, GPU-A3, etc.; GPU-B type GPUs can include GPU-B1, GPU-B2, GPU-B3, etc.; GPU-C type GPUs can include GPU-C1, GPU-C2, GPU-C3, etc.

[0045] It should be noted that Figure 1 The types of processing units and the number of each processing unit in the large language model inference system shown in the figure are only examples. The system can also include more or fewer types of processing units. In addition, each processing unit can include more or fewer numbers of one or more processing units, and there is no limitation on this.

[0046] Each of the multiple processing units can be deployed with a trained large language model to provide inference services to users. Figure 1 In the example, a trained large language model is deployed on each GPU in GPU-A, GPU-B, and GPU-C. Large language models with the same network structure and model parameters can be deployed on different types of processing units. In this case, the large language model can be a processing unit among the various processing units in the large language model inference system (for example, Figure 1 The large language model deployed by the processing unit can be trained by one of the GPUs among GPU-A, GPU-B, and GPU-C in the network, or it can be trained by other processing units (such as other GPU clusters not used for inference, etc.). In this case, the closer the hardware architecture of the processing unit where the trained large language model is deployed is to the processing unit that performed the training, the better the inference effect will be, but the inference cost may be correspondingly higher. On the other hand, multiple processing units can also fine-tune the deployed large language model. At this time, even if different types of processing units deploy large language models with the same network structure, since the model parameters have been fine-tuned, it can also be considered that the large language models deployed by different types of processing units are essentially different.

[0047] Furthermore, each of the multiple processing units can also be used to train a large language model. For example, each of the multiple processing units can train an untrained large language model and use the trained large language model for inference. That is, the type of processing unit performing large language model inference can be the same as the type of processing unit used to train the large language model, thereby ensuring that the processing unit can use the large language model to process inference requests at the theoretical performance during training and generate expected inference results. Furthermore, different types of processing units can be trained and deployed for large language models with the same network structure, or they can be trained and deployed for large language models with different network structures. When different types of processing units train and deploy large language models with the same network structure, although the network structure of the models is the same, the model parameters of the trained models may also be different due to the uncertainty introduced by the training method and different training conditions. Therefore, when each of the multiple processing units trains an untrained large language model and uses the trained large language model for inference, it can be considered that the large language models deployed by the different types of processing units are substantially different. At this point, due to differences in hardware architecture and the large language models deployed, different types of processing units have different inference performance and inference costs. Generally, processing units with better inference performance have higher inference costs, while processing units with relatively poor inference performance have lower inference costs.

[0048] When using a large language model inference system to provide inference services to users, after obtaining an inference request from a user, the inference request can be dispatched to one of the multiple processing units (e.g. Figure 1 In the related art, only the inference effect is usually considered and more inference requests are dispatched to the processing unit with better inference effect. Although this can ensure the quality of the inference result, it will cause the inference cost to be too high. Alternatively, only the inference cost is considered and more inference requests are dispatched to the processing unit with lower inference cost. Although this can reduce the inference cost to a certain extent, it may result in lower quality of the inference result. Alternatively, the inference request is randomly dispatched to any processing unit without considering the inference effect and inference cost. This may be able to control the inference effect and inference cost overall, but it cannot fully utilize the performance or cost advantages of each processing unit. In general, these scheduling methods do not consider user satisfaction to provide users with more cost-effective inference services.

[0049] In order to solve the above technical problems, the present application provides a scheduling method for a large language model inference system with multiple processing units (see below for detailed description). This method can dynamically schedule the user's inference request to one of the multiple processing units by determining the request category to which the user's inference request belongs and based on the scheduling score of each processing unit for the request category, so as to avoid randomly scheduling the user's inference request to any processing unit without considering user satisfaction, or scheduling too many inference requests to processing units with better inference effects, resulting in excessive inference costs, or scheduling too many inference requests to processing units with lower inference costs and ignoring the inference effects, thereby providing users with more satisfactory and cost-effective inference services.

[0050] Exemplarily, the method can be executed by an electronic device such as a terminal device or a server, wherein the terminal device can be a desktop terminal or a mobile terminal, for example, various types of electronic devices such as a laptop computer, a tablet computer, a desktop computer, a smartphone, a smart speaker, a smart watch, a smart TV, and an in-vehicle terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms.

[0051] The following describes in detail a scheduling method for a large language model inference system with multiple processing units provided by this application.

[0052] Figure 2 A flowchart of a scheduling method for a large language model inference system having multiple processing units according to an embodiment of the present disclosure is shown. Different types of processing units in the multiple processing units have different hardware architectures, and each of the multiple processing units has a trained large language model deployed on it. For example, the large language model inference system can be the above Figure 1 The system shown, various processing units may include Figure 1 There are multiple GPUs including GPU-A, GPU-B, GPU-C, etc.

[0053] like Figure 2 As shown, the method may include:

[0054] Step 201: Obtain a user's inference request, where the inference request includes text input by the user.

[0055] Exemplarily, the large language model inference system can obtain an inference request triggered by a user through an interactive interface. For example, a user can input text (prompt) in the form of voice or manual input through an interactive interface on a display screen, and such text (or other information) is obtained by the large language model inference system as an inference request. The content of the text can be any content to which the user hopes the large language model inference system will respond through inference, such as "What is artificial intelligence", "The weather is so good today, and I am in a good mood", etc. It is understandable that each text input by each user can be obtained by the large language model inference system as an inference request. The user's inference request obtained in step 201 can be one or more inference requests from one or more users, and this step and subsequent steps are applicable to each inference request from each user.

[0056] For example, the large language model inference system can pre-process the acquired user's inference request. For example, it can convert the data format of the text in the user's inference request, configure a globally unique identifier for the user's inference request to distinguish each inference request, and so on.

[0057] Step 202: Classify the inference request according to the content of the text using a classification model to determine the request category to which the inference request belongs in at least one preset request category.

[0058] For example, the classification model can be set according to needs and is not limited to this. For example, the classification model can be a bidirectional encoder representation model from transformers (BERT), recurrent neural networks (RNN), convolutional neural networks (CNN), long short-term memory networks (LSTM), transformer models (Transformer), capsule networks (Capsule Networks), graph neural networks (GNNs), Siamese networks (Siamese Networks), hybrid models (Hybrid Models), FastText models, etc. Among them, BERT is a pre-trained deep learning model that can capture deep semantic information in text. It can generate deep feature representations of text, making texts with similar meanings closer in the feature space; RNN can capture the dependencies between words in the text and can understand the time series information in the text; CNN is used in text classification tasks and can identify and extract local features such as keywords or phrases; LSTM can learn long-term dependencies in text and is suitable for processing and classifying longer text sequences; Transformer relies on the self-attention mechanism and can handle long-distance dependency problems and is suitable for capturing global information of text; capsule networks can be used in text classification tasks, especially in multi-label classification tasks; GNNs can encode the syntactic structure in text sentences and is suitable for text classification tasks, especially when processing structured text data; Siamese network is a neural network used for text matching and is suitable for classification tasks; the hybrid model combines attention mechanism, RNN, CNN, etc. to capture local and global features of text; the FastText model can be used for text classification tasks. Among them, the training data of the classification model can be the text of the reasoning request obtained in the past or manually written, and the training process can be implemented using the existing method for the corresponding model, without limitation, as long as the trained classification model has the ability to classify the user's reasoning request according to the content of the text of the user's reasoning request (for example, when the text of the user's reasoning request is input into the model, the model can output the request category of the reasoning request).

[0059] The at least one preset request category can be set based on demand and is not limited to this. As an example, it can be set according to the nature of the text content. In this case, the at least one preset request category may include question category, instruction category, statement category, etc. Among them, the question category indicates that the user wants to obtain information or answer questions, such as "What is artificial intelligence?", "What will the weather be like in Beijing tomorrow?", "How to treat a cold?", etc.; the instruction category indicates that the user wants to obtain specific content, such as "Help me write an article about environmental protection," "List 10 healthy breakfast suggestions," etc.; the statement category indicates that the user expresses opinions, emotions, or states facts, such as "The weather is great today and I'm in a good mood," "I just watched a movie and it was wonderful," etc. As another example, it can be set according to the user's intention reflected by the text content. In this case, the at least one preset request category may include information query category, writing or creation category, advice or guidance category, emotional communication category, etc. Among them, the information query category indicates that users want to obtain specific knowledge or information, such as "What is the history of artificial intelligence?", "What are the recent science and technology news?"; the writing or creation category indicates that users want to generate specific text content, such as "Help me write an article about environmental protection.", "Generate a science fiction story."; the advice or guidance category indicates that users want to obtain advice, solutions or guidance, such as "I have been under a lot of pressure at work recently. Is there any way to relieve it?", "I want to learn a new skill. Do you have any recommendations?"; the emotional communication category indicates that users want to express emotions or engage in emotional interaction, such as "I am in a bad mood today.", "I feel very lonely." and so on.

[0060] For example, assume that at least one preset request category includes three request categories: X, Y, and Z (e.g., X, Y, and Z represent question, instruction, and statement, respectively). In this case, when the large language model inference system obtains the text of the user's inference request "How to cure a cold?" in step 201, it inputs this text into the classification model in step 202. The classification model classifies the user's inference request based on the content of the text, i.e., "How to cure a cold?", and determines that the inference request belongs to request category X (question), Y (instruction), and Z (statement) among the three preset request categories: X (question), Y (instruction), and Z (statement).

[0061] Step 203: Obtain a scheduling score for the request category for each of the multiple processing units, wherein the scheduling score for the request category for each processing unit is based on a satisfaction score for the request category of the processing unit, and the satisfaction score is calculated based on a user rating of a processing result obtained by the processing unit previously processing an inference request for the request category.

[0062] For example, the above-mentioned preset at least one request category includes three request categories X, Y, and Z, and in step 202, the classification model is used to determine that the request category to which the user's reasoning request belongs is X. Assume that the large language model reasoning system is Figure 1 The large language model inference system includes three GPUs, GPU-A, GPU-B, and GPU-C, as three processing units. Since the classification model is used in step 202 to determine that the user's inference request belongs to the request category X, in step 203, the scheduling scores SC of the three processing units GPU-A, GPU-B, and GPU-C for the request category X are obtained respectively. AX , SC BX , SC CX .

[0063] In one possible implementation, a database can be pre-established that includes multiple user ratings of the processing results obtained by each processing unit in the large language model inference system for processing inference requests for each request category within at least one preset request category, a satisfaction score for each processing unit for each request category, and a scheduling score for each processing unit for each request category. Thus, after determining the request category to which the user's inference request belongs using the classification model in step 202, the scheduling score for each processing unit for that request category can be retrieved from the database.

[0064] Exemplarily, when establishing or updating the above-mentioned database, the satisfaction score of each processing unit for each request category can be calculated based on the user rating of the processing results obtained by each processing unit when previously processing the inference requests of each request category, and the scheduling score of each processing unit for each request category can be obtained based on the satisfaction score of each processing unit for each request category.

[0065] User ratings reflect user satisfaction with the processing results of each inference request processed by the large language model inference system. High user ratings generally indicate high user satisfaction, with users believing the results are accurate, useful, and meet their expectations. Low user ratings may indicate low user satisfaction, with users believing the results are unsatisfactory or fail to meet their needs.

[0066] For example, user ratings can be expressed in a percentage, ten-point, five-point, binary, or other format. For example, a numerical value in the range of 0 to 100 can be set to represent the user rating, with 0 representing very dissatisfied and 100 representing very satisfied. For another example, an integer in the range of 1 to 5 can be set to represent the user rating, with 1 representing very dissatisfied and 5 representing very satisfied. For another example, the numerical values 0 and 1 can be set to represent the user rating, with 1 representing satisfied and 0 representing dissatisfied.

[0067] The satisfaction score of each processing unit for each request category is calculated based on the user rating of the processing results obtained by the processing unit when it previously processed the inference request of the request category, so that the satisfaction score of each processing unit for each request category can reflect the user's satisfaction with the processing results obtained by each processing unit when processing the inference request of each request category.

[0068] In one possible implementation, the satisfaction score of each processing unit among the multiple processing units for each request category is the average of user ratings of one or more processing results obtained by the processing unit in processing one or more inference requests of the request category.

[0069] Exemplarily, the satisfaction score of each processing unit for each request category in the large language model reasoning system is the average of the user ratings of multiple processing results obtained by the processing unit previously (for example, within a fixed time period) for processing multiple inference requests of the request category. Different users may have different concerns when scoring the processing results of the inference requests. For example, some users may care about the accuracy of the processing results of the inference requests, some users may care about the inference speed, and so on. Therefore, the user ratings of multiple processing results can be comprehensively considered to calculate the satisfaction score. For example, for each processing unit and each request category in the large language model reasoning system, the average of the user ratings of the multiple processing results obtained by the processing unit previously processing multiple inference requests of the request category is calculated as the satisfaction score of the processing unit for the request category. In this way, the influence of individual evaluation bias can be effectively reduced, so that the satisfaction score of each processing unit for each request category can more objectively reflect the true satisfaction of the user group.

[0070] After calculating the satisfaction score of each processing unit for each request category based on the user rating of the processing results obtained by each processing unit for previously processing the inference request of each request category, the scheduling score of each processing unit for each request category can be obtained and stored based on the satisfaction score (for example, stored in the above-mentioned database).

[0071] For example, the scheduling score of each processing unit for each request category is positively correlated with the satisfaction score of that processing unit for that request category. That is, the higher the satisfaction score of each processing unit for each request category, the higher the scheduling score of that processing unit for that request category; conversely, the lower the satisfaction score of each processing unit for each request category, the lower the scheduling score of that processing unit for that request category.

[0072] In a possible implementation, the scheduling score of each processing unit for each request category may be equal to the satisfaction score of the processing unit for the request category.

[0073] Since the satisfaction score of each processing unit for each request category can reflect the user's rating (satisfaction level) of the processing results obtained for the inference request of that request category, when the satisfaction score of each processing unit for each request category is directly used as the scheduling score of the processing unit for that request category, the user satisfaction with the inference results obtained by subsequent processing of the user's inference request can be guaranteed.

[0074] Because there is a risk of overload when the resource utilization of each processing unit is too high, and there is a risk of resource waste when the resource utilization of each processing unit is too low, the resource utilization of the processing unit may be considered in addition to user satisfaction when calculating the scheduling score. In one possible implementation, the scheduling score of each processing unit for each request category is calculated by multiplying the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit by preset weights, and then summing the results. The weighted sum of the satisfaction score and the resource utilization is calculated based on the number of processing units in use and the total number of processing units of that type.

[0075] For example, for each type of processing unit, among all the processing units of this type in the large language model inference system, the number of processing units currently performing inference services, i.e., the number of used processing units, is determined, and the ratio of the number of used processing units to the total number of processing units of this type in the large language model inference system is calculated as the resource utilization rate of this type of processing unit. Figure 1 Taking the language model inference system of Zhongda University as an example, if there are N GPU-C GPUs deployed in the system (i.e., GPU-C1, GPU-C2, GPU-C3...GPU-CN), among which, at the current moment, the number of GPUs performing inference services among these N GPU-C GPUs is M (M≤N), then the resource utilization of GPU-C is M / N.

[0076] After obtaining the satisfaction score and resource utilization, the scheduling score (SC) can be calculated using the formula SC = w1*SS+w2*RR. SS is the satisfaction score for each processing unit for each request category, RR is the resource utilization for that processing unit, and w1 and w2 are the weights assigned to that processing unit for the satisfaction score and resource utilization, respectively. For each processing unit, the specific values for the weight w1 of the satisfaction score for each request category and the weight w2 of the resource utilization for that processing unit can be set as needed and are not limited. It should be understood that a greater weight indicates a greater weighting of the corresponding item when calculating the scheduling score. For example, a greater weight w1 for the satisfaction score indicates a greater emphasis on user satisfaction when calculating the scheduling score, while a greater weight w2 for resource utilization indicates a greater emphasis on resource utilization when calculating the scheduling score. These two weights can also be dynamically adjusted based on the changing trends of the satisfaction score, as described below. Furthermore, the weights assigned to the satisfaction scores of different processing units can be the same or different. Similarly, the weights of resource utilization for different types of processing units may be the same or different.

[0077] Assume that the large language model inference system is Figure 1 The large language model inference system in

[15] includes three GPUs: GPU-A, GPU-B, and GPU-C as three processing units. It is assumed that the satisfaction scores of GPU-A for the three request categories X, Y, and Z are SS and SS, respectively. AX , SS AY , SS AZ , GPU-B’s satisfaction scores for the three request categories X, Y, and Z are SS and BX , SS BY , SS BZ , GPU-C’s satisfaction scores for the three request categories X, Y, and Z are SS and CX , SS CY , SS CZ , and the resource utilization of GPU-A, GPU-B, and GPU-C at the current moment are RR A , RR B , RR C Assuming that the weight of each processing unit for satisfaction score and resource utilization is w1 and w2, the scheduling score SC of GPU-A for the three request categories X, Y, and Z is AX , SC AY , SC AZ SC AX =w1*SS AX +w2*RR A , SCAY =w1*SS AY +w2*RR A , SC AZ =w1*SS AZ +w2*RR A , GPU-B's scheduling score SC for the three request categories X, Y, and Z BX , SC BY , SC BZ SC BX =w1*SS BX +w2*RR B , SC BY =w1*SS BY +w2*RR B , SC BZ =w1*SS BZ +w2*RR B , GPU-C scheduling score SC for the three request categories X, Y, and Z CX , SC CY , SC CZ SC CX =w1*SS CX +w2*RR C , SC CY =w1*SS CY +w2*RR C , SC CZ =w1*SS CZ +w2*RR C The case where the scheduling score of each processing unit for each request category is equal to the satisfaction score can be regarded as a special case when w1=1 and w2=0.

[0078] In this way, for each processing unit, the weighted sum of the satisfaction score and the resource utilization rate of the processing unit for each request category is calculated by multiplying the satisfaction score of the processing unit for each request category and the resource utilization rate of the processing unit by the preset weights and summing them up as the scheduling score of the processing unit for the request category. This can comprehensively consider the user's satisfaction and the resource utilization rate of the processing unit when scheduling inference requests. While ensuring user satisfaction, it can balance the loads of different types of processing units as much as possible and try to avoid the situation where a certain processing unit is idle or overloaded.

[0079] Step 204: Schedule the inference request to one of the multiple processing units based on the scheduling score of each of the multiple processing units for the request category, so that the large language model deployed by the one processing unit processes the inference request.

[0080] The scheduling probability can be calculated based on the scheduling score, such that the higher the scheduling score of each processing unit for each request category, the higher the probability that an inference request of that request category will be scheduled to that processing unit; and the lower the scheduling score of each processing unit for each request category, the lower the probability that an inference request of that request category will be scheduled to that processing unit. Thus, the scheduling score is associated with the scheduling probability. Furthermore, because the scheduling score of each processing unit for the request category to which the user's inference request belongs is based on a satisfaction score that reflects the user's previous satisfaction with the processing results obtained by each processing unit for processing inference requests of that request category, the user's inference request can be scheduled to one of multiple processing units based on the user's satisfaction. Furthermore, rather than mechanically selecting the processing unit with the highest scheduling score to process the user's inference request, the scheduling score is used as the scheduling probability. While the processing unit with the highest scheduling score is most likely to process the inference request, other types of processing units still have the possibility of being scheduled, thereby enabling the comprehensive utilization of different types of processing units. In this way, it is possible to avoid randomly scheduling users' inference requests to any processing unit without considering user satisfaction, or scheduling too many inference requests to processing units with better inference effects, resulting in excessively high inference costs, or scheduling too many inference requests to processing units with lower inference costs and ignoring the inference effects, thereby providing users with inference services with higher satisfaction and cost-effectiveness.

[0081] In one possible implementation, scheduling the inference request to one of the multiple processing units based on the scheduling score of each of the multiple processing units for the request category includes: taking the ratio of the scheduling score of each processing unit for the request category to the total scheduling score of the multiple processing units for the request category as the scheduling probability of the processing unit for the request category, and scheduling the inference request to one of the multiple processing units according to the scheduling probability of each of the multiple processing units for the request category. The total scheduling score of the multiple processing units for the request category to which the user's inference request belongs is the sum of the scheduling scores of each of the multiple processing units for the request category.

[0082] For example, when the scheduling score is expressed in a numerical value of 100, 10, 5, or 2, the total scheduling score is calculated as the sum of the scheduling scores of each processing unit for the request category to which the user's inference request belongs. Figure 1The large language model inference system in the example includes three GPUs, GPU-A, GPU-B, and GPU-C, as three processing units. In step 202, the classification model is used to determine that the request category X to which the user's inference request belongs. In step 203, the scheduling scores SC of the three processing units, GPU-A, GPU-B, and GPU-C, for the request category X are obtained respectively. AX , SC BX , SC CX At this time, the total scheduling score of the three processing units for request category X is SC AX +SC BX +SC CX Then, for each processing unit, the ratio of the scheduling score of the processing unit for the request category to which the user's inference request belongs to the total scheduling score is calculated as the scheduling probability of the processing unit for the request category. It can be calculated that the scheduling probabilities of the three processing units GPU-A, GPU-B, and GPU-C for request category X are SC and SC, respectively. AX / (SC AX +SC BX +SC CX ), SC BX / (SC AX +SC BX +SC CX ), SC CX / (SC AX +SC BX +SC CX ). The user's inference request can be dispatched to a processing unit according to such probability.

[0083] When the ratio of the scheduling score of each processing unit for the request category to which the user's inference request belongs to the total scheduling scores of the multiple processing units for the request category is used as the scheduling probability of the processing unit for the request category, and the inference request is scheduled to one of the multiple processing units according to the scheduling probability of each processing unit in the multiple processing units for the request category, specifically, for example, an array consisting of different numbers can be pre-constructed, wherein the different numbers represent different types of processing units, and the number of each number in the array is proportional to the scheduling score of the processing unit corresponding to the number for the request category to which the user's inference request belongs, and each number in the array is numbered. For example, there are three types of processing units in the large language model inference system, and an array consisting of three numbers can be pre-constructed, and the number of the three numbers in the array is the same as the scheduling score of the corresponding type of processing unit for the request category to which the user's inference request belongs, and each number in the array is numbered. Furthermore, during actual scheduling, a random number generation function can be used to randomly generate a number within a numerical range consisting of the numbers in the array, and the user's inference request can be dispatched to the processing unit of the type corresponding to the number in the array. In this way, the probability of each number in the array being selected is the same as the scheduling probability of the corresponding type of processing unit for the request category to which the user's inference request belongs. This allows the user's inference request to be dispatched to one of the multiple processing units based on the scheduling probability of each processing unit for the request category to which the user's inference request belongs.

[0084] As an example, take the above Figure 1 Taking the language model inference system of Zhongda University as an example, if the system includes three GPUs: GPU-A, GPU-B, and GPU-C as three processing units, and the scheduling scores SC of GPU-A, GPU-B, and GPU-C for request category X AX , SC BX , SC CXThe total scheduling score is 30, 50, and 60, respectively. The total scheduling score is 30 + 50 + 60 = 140. The scheduling probabilities of the three processing units for request category X are 30 / 140, 50 / 140, and 60 / 140, respectively. For example, the number 0 can be used to represent GPU-A, the number 1 to represent GPU-B, and the number 2 to represent GPU-C. 30 "0s," 50 "1s," and 60 "2s" can be used to form an array of 140 numbers. The different numbers in this array are sequentially numbered from 0 to 139. For example, the 30 "0s" are sequentially numbered from 0 to 29; the 50 "1s" are sequentially numbered from 30 to 79; and the 60 "2s" are sequentially numbered from 80 to 139. In practical applications, a random number generator function can be used to randomly generate a number K from 0 to 139, and the user's inference request can be scheduled to the GPU type corresponding to the number K. For example, if K=21 is randomly generated and the number 21 is "0", the user's inference request will be dispatched to GPU-A represented by "0". For another example, if K=100 is randomly generated and the number 100 is "2", the user's inference request will be dispatched to GPU-C represented by "2". In this way, the probability that the randomly generated K from 0 to 139 falls within the range of 0 to 29 (representing GPU-A's "0") is 30 / 140, the probability that it falls within the range of 30 to 79 (representing GPU-B's "1") is 50 / 140, and the probability that it falls within the range of 80 to 139 (representing GPU-C's "2") is 60 / 140. These are the same as the scheduling probabilities of 30 / 140, 50 / 140, and 60 / 140 for GPU-A, GPU-B, and GPU-C for request category X, respectively. This allows the user's inference request to be scheduled to one of GPUs A, B, and C according to their respective scheduling probabilities for request category X.

[0085] As another implementation method for dispatching an inference request to one of multiple processing units based on the scheduling probability of each processing unit for the request category to which the user's inference request belongs, an array consisting of numbers in different numerical ranges can be pre-constructed. Different numerical ranges represent different types of processing units, and the size of each numerical range is proportional to the scheduling score of the processing unit corresponding to that numerical range for the request category to which the user's inference request belongs. For example, for three types of processing units, an array consisting of three numerical ranges can be pre-constructed, with the size of each numerical range being the same as the scheduling score of the corresponding type of processing unit for the request category to which the user's inference request belongs. Furthermore, during actual scheduling, a random number generation function can be used to randomly generate a number in the array, and the user's inference request is dispatched to the processing unit of the type corresponding to the numerical range in which the number falls. In this way, the probability of each number in the array being selected is the same as the scheduling probability of the processing unit of the type corresponding to the numerical range in which the number falls for the request category to which the user's inference request belongs, thereby dispatching the user's inference request to one of the multiple processing units based on the scheduling probability of each processing unit for the request category to which the user's inference request belongs.

[0086] As an example, take the above Figure 1 Taking the language model inference system of Zhongda University as an example, the system includes three types of GPUs: GPU-A, GPU-B, and GPU-C as three processing units, and the scheduling scores SC of GPU-A, GPU-B, and GPU-C for request category X AX , SC BX , SC CXThe values of K and K are 30, 50, and 60, respectively. The total scheduling score is 30 + 50 + 60 = 140. The scheduling probabilities of the three processing units for request category X are 30 / 140, 50 / 140, and 60 / 140, respectively. For example, the numerical range 0-29 can be used to represent GPU-A, the numerical range 30-79 to represent GPU-B, and the numerical range 80-139 to represent GPU-C. The numbers in these three numerical ranges 0-139 are then combined into an array, i.e., the array includes the numbers 0-139. In practical applications, a random number generation function can be used to randomly generate a number K between 0 and 139, and the user's inference request is scheduled to the GPU type corresponding to the numerical range of K. For example, if K is randomly generated to be 21, the user's inference request is scheduled to GPU-A, which is represented by the numerical range 0-29. For another example, if K is randomly generated to be 100, the user's inference request is scheduled to GPU-C, which is represented by the numerical range 80-139. In this way, the probability that K randomly generated from 0 to 139 falls within the numerical range of 0 to 29 representing GPU-A is 30 / 140, the probability that it falls within the numerical range of 30 to 79 representing GPU-B is 50 / 140, and the probability that it falls within the numerical range of 80 to 139 representing GPU-C is 60 / 140. These are the same as the scheduling probabilities of GPU-A, GPU-B, and GPU-C for request category X, which are 30 / 140, 50 / 140, and 60 / 140, respectively. Therefore, the user's inference request is scheduled to one of GPU-A, GPU-B, and GPU-C according to the scheduling probabilities of GPU-A, GPU-B, and GPU-C for request category X.

[0087] After the inference request is dispatched to one of the multiple processing units in the above manner, one or more processing units of this type of processing unit can process the user's inference request through data parallelism, model parallelism, etc., thereby efficiently utilizing each processing unit of this type of processing unit, avoiding the situation where some processing units are overloaded while other processing units are idle, and improving the performance of the system.

[0088] In an embodiment of the present disclosure, through the above steps 201 to 204, a user's inference request is obtained, and the inference request includes a text input by the user; the inference request is classified according to the content of the text using a classification model to determine the request category to which the inference request belongs in at least one preset request category; a scheduling score for each of the multiple processing units for the request category is obtained, wherein the scheduling score for each of the multiple processing units for the request category is based on a satisfaction score of the processing unit for the request category, and the satisfaction score is calculated based on a user rating of a processing result obtained by the processing unit previously processing the inference request of the request category; based on the scheduling score of each of the multiple processing units for the request category, the inference request is scheduled to one of the multiple processing units, so that the large language model deployed by the one processing unit processes the inference request. Therefore, by determining the request category to which the user's reasoning request belongs and based on the scheduling score of each processing unit for the request category, the user's reasoning request can be dynamically scheduled to one of the multiple processing units, so as to avoid randomly scheduling the user's reasoning request to any processing unit without considering user satisfaction, or scheduling too many reasoning requests to processing units with better reasoning effects, resulting in excessively high reasoning costs, or scheduling too many reasoning requests to processing units with lower reasoning costs and ignoring the reasoning effects, thereby providing users with reasoning services with higher satisfaction and cost-effectiveness.

[0089] Figure 3 FIG. 1 is a flowchart showing another scheduling method for a large language model inference system with multiple processing units according to an embodiment of the present disclosure. Figure 3 As shown, the method may include:

[0090] Step 301: Obtain a user's inference request, where the inference request includes text input by the user.

[0091] Step 302: Using a classification model, classify the inference request according to the content of the text to determine the request category to which the inference request belongs in at least one preset request category.

[0092] Step 303: Obtain a scheduling score for the request category for each of the multiple processing units, wherein the scheduling score for the request category for each processing unit is based on a satisfaction score for the request category of the processing unit, and the satisfaction score is calculated based on a user rating of a processing result obtained by the processing unit previously processing an inference request for the request category.

[0093] Step 304: Schedule the inference request to one of the multiple processing units based on the scheduling score of each of the multiple processing units for the request category, so that the large language model deployed by the one processing unit processes the inference request.

[0094] The above steps 301 to 304 are the same as the above Figure 2 Steps 201 to 204 are the same and will not be repeated here.

[0095] Step 305: After the large language model deployed in the processing unit processes the inference request to obtain a processing result, obtain a user rating of the processing result.

[0096] For example, after the large language model deployed by the processing unit scheduled in the large language model inference system processes the user's inference request and obtains the processing result, the processing result can be presented to the user, for example, it can be displayed to the user through an interactive interface of a display screen; accordingly, a prompt message can be sent to the user to prompt the user to evaluate the processing result, thereby obtaining the user's user rating of the processing result.

[0097] As an example, a star rating button group can be set on the interactive interface. For example, there can be options of 1-5 stars, and the user can click the corresponding number of stars according to the satisfaction with the processing results, where 1 star represents very dissatisfied and 5 stars represent very satisfied. As another example, a horizontal or vertical slider can be set on the interactive interface, with one end representing "very dissatisfied" and the other end representing "very satisfied", and the user can select a specific satisfaction position by dragging the slider. Exemplarily, the slider can also have a scale from 0 to 100, with 0 representing very dissatisfied and 100 representing very satisfied. The user can drag the slider to the appropriate position according to his or her satisfaction with the processing results, and the system will record the corresponding score according to the position of the slider.

[0098] In one possible implementation, to encourage users to rate, some guidance can be provided on the interactive interface. For example, after the interactive interface displays the processing results to the user, a prompt box can automatically pop up: "We would love to hear your feedback. Please rate this processing result!" or "Evaluate to receive rewards. Rate this processing result now."

[0099] Step 306: Store the user rating in association with the type of the processing unit and the request category to which the inference request belongs.

[0100] Exemplarily, the user's rating of the current processing result, the type of the processing unit that processes the user's reasoning request, and the request category to which the user's reasoning request belongs may be associated and stored in the pre-built database.

[0101] In an embodiment of the present disclosure, after the user's inference request is dispatched to one of a plurality of processing units and the large language model deployed thereon processes the inference request to obtain a processing result, the user's rating of the processing result is obtained, and the user rating, the type of processing unit scheduled this time, and the request category to which the user's inference request belongs are stored in association with each other. This allows for dynamic and continuous updating of data in the database, providing a data basis for calculating the satisfaction score of each processing unit for each request category and the corresponding scheduling score.

[0102] Furthermore, considering that the number of inference requests in a large language model inference system is usually large, various processing units in the large language model inference system will process a large number of inference requests of different request categories. As user ratings of the processing results obtained for each inference request are continuously obtained, the data in the above database can be updated.

[0103] In a possible implementation, the method further includes: updating, at fixed time intervals, a satisfaction score of each processing unit among the multiple processing units for each request category in the at least one request category.

[0104] Exemplarily, the satisfaction score of each processing unit for each request category stored in the above database can be updated based on the user ratings obtained from the processing results of the inference requests of each request category processed by multiple processing units within a fixed time period (e.g., one day, three days, one week, etc.), thereby enabling dynamic updating of the satisfaction score. The initial values of the satisfaction scores of the various processing units for each request category in the database can be set to the same value. For example, taking the satisfaction score as a percentage as an example, the initial value of the satisfaction score of each processing unit for each request category in the large language model inference system can be set to 100, and at every fixed time period, the satisfaction score can be updated based on the multiple user ratings of the processing results obtained by each processing unit processing the inference requests of each request category within the fixed time period. As a result, the satisfaction score of each processing unit for each request category in the database will continue to change accordingly.

[0105] Exemplarily, updating the satisfaction score of each of the multiple processing units for each of the at least one request category includes: obtaining user ratings of one or more processing results obtained by each of the multiple processing units processing one or more inference requests for each of the at least one request category within the fixed time period; and updating, for each request category, the satisfaction score of each processing unit for the request category to the average of the user ratings of the one or more processing results obtained by each processing unit processing the one or more inference requests for the request category within the fixed time period. In this way, the satisfaction scores of various processing units for each request category can be continuously updated as users use the large language model inference system to perform inference and evaluate the processing results.

[0106] As an example, for each request category, user ratings of the processing results obtained by each processing unit for all inference requests of that request category processed within the fixed time period can be calculated. Furthermore, for each processing unit, the average user rating of the processing results obtained by that processing unit for all inference requests of that request category processed within the fixed time period is calculated. That is, the ratio of the sum of the user ratings of the processing results obtained by that processing unit for all inference requests of that request category processed within the fixed time period to the number of inference requests of that request category processed by that processing unit within the fixed time period is calculated, and this average value is used as the latest satisfaction score of that processing unit for that request category.

[0107] As another example, at every fixed time period, for each request category, the user rating of the processing results obtained by each processing unit for all inference requests of the request category processed within the fixed time period can be obtained. Then, for each processing unit, the user rating of the processing results obtained by this processing unit for all inference requests of the request category processed within the fixed time period and the user rating of the processing results obtained by this processing unit for all inference requests of the request category processed within the historical fixed time period are averaged to obtain the average value of the user ratings of the processing results obtained by this processing unit for processing the inference requests of the request category within all fixed time periods (i.e., this fixed time period and the historical fixed time period). The average value is then used as the latest satisfaction score of this processing unit for the request category.

[0108] In one possible implementation, the method further includes updating, at regular intervals, a scheduling score of each of the various processing units for each of the at least one request category based on the updated satisfaction score of each of the multiple processing units for each of the at least one request category. That is, the scheduling score is updated as the satisfaction score is updated.

[0109] Exemplarily, the scheduling scores of each processing unit in the large language model inference system stored in the above-mentioned database for each preset request category can be updated. In this way, after obtaining the latest inference request, the large language model can determine the request category to which the inference request belongs, and then obtain the latest scheduling score of each processing unit among the multiple processing units for the request category, and then, based on the latest scheduling score of each processing unit for the request category, schedule the inference request to one of the multiple processing units, so that the large language model deployed by the processing unit of this type processes the latest inference request. Since the latest scheduling score of each processing unit for the request category is obtained based on the updated satisfaction score, and the updated satisfaction score can reflect the user's latest satisfaction with the processing results obtained by each processing unit for processing the inference request of the request category, the user's inference request can be scheduled to a processing unit for processing considering the user's latest satisfaction.

[0110] With the above Figure 1 Take the large language model inference system as an example, where three GPUs, GPU-A, GPU-B, and GPU-C, are used as three processing units. Assume that at least one preset request category includes request category X, request category Y, and request category Z. The database stores user ratings of the processing results obtained by GPU-A, GPU-B, and GPU-C for processing inference requests of request category X, request category Y, and request category Z, the satisfaction score of each GPU for each request category, and the scheduling score of each GPU for each request category. To simplify the explanation, it is assumed that the satisfaction score of each GPU for each request category and the scheduling score of each GPU for each request category are configured to be the same, and the initial values of the satisfaction score of each GPU for each request category and the scheduling score of each GPU for each request category are both set to 100. As the large language model inference system continues to provide inference services, the user ratings of the processing results obtained by GPU-A, GPU-B, and GPU-C for processing inference requests of the three request categories X, Y, and Z can be obtained at fixed time intervals, and the satisfaction score of each GPU for each request category is updated based on the obtained user ratings. For example, at midnight on a certain night, for request category X, the user ratings and average values obtained by GPU-A, GPU-B, and GPU-C for processing inference requests of request category X from midnight to midnight on the same day are shown in the following table.

[0111]

[0112] Therefore, the satisfaction scores of GPU-A, GPU-B, and GPU-C for request category X can be updated to the average user ratings of 76, 95, and 90 obtained from processing inference requests for request category X, respectively. Since the scheduling score is equal to the satisfaction score, the scheduling scores of GPU-A, GPU-B, and GPU-C for request category X can also be updated to 76, 95, and 90, respectively. Thus, between 0:00 and 24:00 on the next day, based on the updated scheduling scores of 76, 95, and 90 for request category X obtained by GPU-A, GPU-B, and GPU-C, inference requests for request category X can be scheduled to one of the processing units among GPU-A, GPU-B, and GPU-C for processing. At 24:00 on the next day, the satisfaction scores and scheduling scores can be updated based on the user ratings of the next day. Similar updates can be performed for request categories Y and Z. It should be understood that the number and specific values of user ratings in the above examples are merely simplified examples for ease of understanding. In actual applications, there may be more user ratings for each request category within a preset time period, and the user ratings can be any value within a preset range.

[0113] Furthermore, the method may also include: analyzing the changing trend of the satisfaction score based on the satisfaction score of each processing unit among the multiple processing units for each request category in the at least one request category within a plurality of consecutive fixed time periods; when the scheduling score of each processing unit for the request category is the weighted sum of the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit, adjusting the respective preset weights of the satisfaction score and the resource utilization when calculating the weighted sum based on the changing trend of the satisfaction score.

[0114] For example, the initial values of the preset weights w1 and w2 of the satisfaction score and the resource utilization rate, respectively, may both be 0.5, that is, the initial values of the preset weights of the two are the same.

[0115] Exemplarily, for each request category in at least one preset request category, a statistical analysis can be performed on the satisfaction score of each processing unit in a plurality of processing units for the request category within a plurality of consecutive fixed time periods, thereby analyzing the changing trend of the satisfaction score of each processing unit for the request category; for example, analysis can be performed using existing statistical analysis methods such as descriptive statistical analysis and regression analysis to obtain the changing trend of the satisfaction score.

[0116] Exemplarily, for each request category, if it is analyzed that the satisfaction score of a certain processing unit for the request category shows a downward trend within multiple consecutive fixed time periods, it indicates that the user's evaluation of the processing results obtained by this processing unit for processing the inference request of this request category is getting worse. Therefore, when calculating the weighted sum of the satisfaction score of this processing unit for the request category and the resource utilization rate of this processing unit as the scheduling score of this processing unit for the request category, the preset weight w1 corresponding to the satisfaction score can be increased, and / or the preset weight w2 corresponding to the resource utilization rate can be reduced, so as to increase the relative proportion of the satisfaction score and pay more attention to the user's satisfaction (or inference effect). Alternatively, for each request category, if, over multiple consecutive fixed time periods, it is analyzed that the satisfaction score of a processing unit with a higher inference cost for that request category shows an upward trend, this indicates that the user's evaluation of the processing results obtained by that processing unit for processing inference requests of that request category is improving, but the inference cost is correspondingly increasing. Therefore, when calculating the weighted sum of the satisfaction score of that processing unit for that request category and the resource utilization rate of that processing unit as the scheduling score for that processing unit for that request category, the preset weight w1 corresponding to the satisfaction score can be appropriately reduced, and / or the preset weight w2 corresponding to the resource utilization rate can be increased to increase the relative proportion of resource utilization and pay more attention to resource utilization (or inference cost). In this way, by multiplying the satisfaction score of that processing unit for that request category and the resource utilization rate of that processing unit by the adjusted preset weights and summing them to calculate the weighted sum of the satisfaction score and resource utilization rate as the scheduling score, the scheduling score calculation method can be timely adjusted according to the changing trend of user satisfaction, thereby improving user satisfaction with the processing results of the inference request while maintaining a balance between inference effect and inference cost.

[0117] Based on the same inventive concept of the above method embodiment, an embodiment of the present disclosure also provides a scheduling device for a large language model inference system with multiple processing units, which can be used to execute the technical solution described in the above method embodiment.

[0118] Figure 4 A structural diagram of a scheduling device for a large language model inference system having multiple processing units according to an embodiment of the present disclosure is shown. Different types of processing units in the multiple processing units have different hardware architectures, and each of the multiple processing units is deployed with a trained large language model, such as Figure 4As shown, the device includes: an acquisition module 401, which is used to obtain a user's inference request, and the inference request includes a text input by the user; a classification module 402, which is used to use a classification model to classify the inference request according to the content of the text to determine the request category to which the inference request belongs in at least one preset request category; the acquisition module 401 is also used to obtain a scheduling score of each of the multiple processing units for the request category, wherein the scheduling score of each processing unit for the request category is based on the satisfaction score of the processing unit for the request category, and the satisfaction score is calculated based on the user rating of the processing result obtained by the processing unit previously processing the inference request of the request category; a scheduling module 403 is used to schedule the inference request to one of the multiple processing units based on the scheduling score of each of the multiple processing units for the request category, so that the large language model deployed by the one processing unit processes the inference request.

[0119] In an embodiment of the present disclosure, a user's inference request is obtained, where the inference request includes text input by the user; a classification model is used to classify the inference request according to the content of the text to determine the request category to which the inference request belongs in at least one preset request category; a scheduling score for each of the multiple processing units for the request category is obtained, wherein the scheduling score for each of the multiple processing units for the request category is based on a satisfaction score of the processing unit for the request category, and the satisfaction score is calculated based on a user rating of a processing result obtained by the processing unit previously processing the inference request of the request category; and based on the scheduling score for the request category of each of the multiple processing units, the inference request is scheduled to one of the multiple processing units, so that the large language model deployed by the one processing unit processes the inference request. In this way, the request category to which the user's inference request belongs is determined, and then, for multiple processing units with different hardware architectures, the user's inference request is scheduled to one of the multiple processing units based on the scheduling score of each processing unit for the request category to which the user's inference request belongs. The scheduling score represents the scheduling probability, thereby realizing dynamic scheduling of different types of processing units, avoiding the problem of poor inference results or high inference costs caused by always scheduling the same type of processing unit, and achieving a balance between inference effect and inference cost. The obtained inference results can meet the quality requirements, and at the same time can reduce the inference cost to a certain extent, thereby providing users with more satisfactory and cost-effective inference services.

[0120] In one possible implementation, the scheduling module 403 is further used to: take the ratio of the scheduling score of each processing unit for the request category to the total scheduling scores of the multiple processing units for the request category as the scheduling probability of the processing unit for the request category, and schedule the inference request to one of the multiple processing units according to the scheduling probability of each processing unit in the multiple processing units for the request category.

[0121] In a possible implementation, the scheduling score of each processing unit for the request category is equal to the satisfaction score of the processing unit for the request category.

[0122] In one possible implementation, the scheduling score of each processing unit for the request category is a weighted sum of the satisfaction score and the resource utilization calculated by multiplying the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit by preset weights and summing the results. The resource utilization is calculated based on the number of processing units used in the processing unit and the total number of processing units of the processing unit.

[0123] In one possible implementation, the satisfaction score of each of the multiple processing units for the request category is the average of user ratings of one or more processing results obtained by the processing unit in processing one or more inference requests of the request category.

[0124] In one possible implementation, the acquisition module 401 is further used to: after the large language model deployed by the one processing unit processes the inference request to obtain a processing result, obtain a user rating of the processing result; and store the user rating in association with the type of the one processing unit and the request category to which the inference request belongs.

[0125] In one possible implementation, the device also includes an update module for: updating the satisfaction score of each of the multiple processing units for each request category in the at least one request category at every fixed time period, and the update includes: obtaining user scores of one or more processing results obtained by each of the multiple processing units processing one or more inference requests for each request category in the at least one request category within the fixed time period; for each request category, updating the satisfaction score of each processing unit for the request category to the average value of the user scores of one or more processing results obtained by each processing unit processing one or more inference requests for the request category within the fixed time period.

[0126] In one possible implementation, the device also includes an adjustment module for: analyzing a changing trend of the satisfaction score based on the satisfaction score of each processing unit among the multiple processing units for each request category in the at least one request category within a plurality of consecutive fixed time periods; and adjusting the preset weights of the satisfaction score and the resource utilization when calculating the weighted sum based on the changing trend of the satisfaction score, when the scheduling score of each processing unit for the request category is a weighted sum of the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit.

[0127] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0128] An embodiment of the present disclosure further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0129] An embodiment of the present disclosure further provides a non-volatile computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0130] An embodiment of the present disclosure further provides a computer program product, including a computer program, or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0131] Figure 5 FIG1 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0132] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0133] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0134] A computer-readable storage medium can be a tangible device that can hold and store programs / instructions used by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0135] The computer programs (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0136] The computer program (or computer program instructions) for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing state information of computer-readable program instructions to personalize and customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present disclosure.

[0137] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0139] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A scheduling method for a large language model inference system with multiple processing units, characterized in that: Different types of processing units among the multiple processing units have different hardware architectures, and a trained large language model is deployed on each of the multiple processing units. The method includes: Obtaining a user's reasoning request, where the reasoning request includes text input by the user; Using a classification model, classifying the inference request according to the content of the text, so as to determine the request category to which the inference request belongs in at least one preset request category; Obtaining a scheduling score for each of the multiple processing units for the request category, wherein the scheduling score for each processing unit for the request category is based on a satisfaction score for the request category obtained by the processing unit, the satisfaction score being calculated based on user ratings of processing results obtained by the processing unit when previously processing inference requests for the request category; The inference request is dispatched to one of the multiple processing units based on a dispatch score of each of the multiple processing units for the request category, so that the large language model deployed by the one processing unit processes the inference request.

2. The method according to claim 1, characterized in that The step of dispatching the inference request to one of the plurality of processing units based on a dispatch score of each of the plurality of processing units for the request category comprises: The ratio of the scheduling score of each processing unit for the request category to the total scheduling scores of the multiple processing units for the request category is used as the scheduling probability of the processing unit for the request category, and the inference request is scheduled to one of the multiple processing units according to the scheduling probability of each processing unit among the multiple processing units for the request category.

3. The method according to claim 2, characterized in that The scheduling score of each processing unit for the request category is equal to the satisfaction score of the processing unit for the request category.

4. The method according to claim 2, characterized in that The scheduling score of each processing unit for the request category is calculated by multiplying the satisfaction score of the processing unit for the request category and the resource utilization rate of the processing unit by a preset weight and summing the results. The resource utilization rate is calculated based on the number of used processing units of the type of processing units and the total number of the type of processing units.

5. The method according to claim 3 or 4, characterized in that The satisfaction score of each processing unit among the multiple processing units for the request category is an average of user ratings of one or more processing results obtained by the processing unit in processing one or more inference requests of the request category.

6. The method according to claim 5, characterized in that The method further comprises: After the large language model deployed in the one processing unit processes the inference request to obtain a processing result, obtaining a user rating of the processing result by a user; The user score is stored in association with the type of the processing unit and the request category to which the inference request belongs.

7. The method according to claim 6, characterized in that The method further comprises: At regular time intervals, updating the satisfaction score of each processing unit in the plurality of processing units for each request category in the at least one request category, the updating comprising: Obtaining user scores of one or more processing results obtained by each of the multiple processing units processing one or more inference requests of each request category in the at least one request category within the fixed time period; For each request category, the satisfaction score of each processing unit for the request category is updated to the average user rating of one or more processing results obtained by each processing unit processing one or more inference requests of the request category within the fixed time period.

8. The method according to claim 7, characterized in that The method further comprises: analyzing a change trend of the satisfaction score according to the satisfaction score of each processing unit of the multiple processing units for each request category of the at least one request category within a plurality of consecutive fixed time periods; When the scheduling score of each processing unit for the request category is the weighted sum of the satisfaction score of the processing unit for the request category and the resource utilization of the processing unit, based on the changing trend of the satisfaction score, adjust the preset weights of the satisfaction score and the resource utilization when calculating the weighted sum.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 8.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images