A resource scheduling method and device for large model reasoning requests

Abstract

The embodiment of the present invention provides a resource scheduling method and device for large model reasoning requests, which relates to the field of computer technology. The method is applied to a scheduling server in a server cluster, and the server cluster also includes multiple target servers, each of which is deployed with multiple large models. The method includes: determining each target reasoning request and the computing resource requirement of each target reasoning request; sorting the computing resource requirement of each target reasoning request, the current remaining resources of each target server and the server performance of each target server, and substituting them into the objective function to determine the allocation result of each target reasoning request; for each target reasoning request, allocating the request to the target server represented by the allocation result, so that the target server performs resource scheduling and processing according to the corresponding computing resource requirement, so as to achieve the purpose of optimizing the usage of computing resources of servers in the server cluster.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a resource scheduling method and device for large model reasoning requests. Background Art

[0002] The emergence of big model technology marks a new era of information processing and intelligent decision-making. These advanced models have unique advantages in understanding and generating semantic content with their powerful data processing capabilities and complex algorithm structures. Whether in text, audio, image, or video data, big models have demonstrated their powerful parsing and generation capabilities. As big models penetrate into all aspects of corporate operations, financial institutions are taking advantage of the advantages of big models to more efficiently sort out and analyze massive amounts of data, thereby improving the speed and accuracy of decision-making, achieving rich semantic understanding of big models, helping companies improve business performance, and gaining advantages in fierce market competition.

[0003] Generally speaking, the big model uses the server where the big model is deployed as the basic computing power facility. In other words, the big model resources required to process each inference request (also called big model inference request) are actually the computing resources provided by the server when the big model processes each inference request on the deployed server. However, the computing resources of the server currently face problems such as high resource costs. For example, the graphics card resources, network bandwidth and other resources of high-performance servers require higher capital costs.

[0004] In the related technology, by building a distributed cluster consisting of multiple servers, sufficient graphics card resources are provided for the operation of large models. In this way, after determining the graphics card resources required for each received inference request, the corresponding graphics card resources are provided for the inference request by scheduling the graphics card resources of the server cluster.

[0005] However, in the case of limited computing resources, in order to ensure user experience, when scheduling resources, it is also necessary to ensure the coordinated scheduling of network resources and graphics card resources, that is, one computing resource cannot be considered alone. However, in the above-mentioned related technologies, only the scheduling of graphics card resources is considered, and the scheduling of other resources, such as the scheduling of network bandwidth resources, is not considered. As a result, when computing resources are limited, if there is a shortage or overflow of a computing resource during the resource scheduling process, the computing resources of the server are lacking or wasted.

[0006] Based on this, how to coordinate the scheduling of graphics card resources and network bandwidth resources of the server cluster under the condition of limited computing resources to reduce the possibility of shortage or waste of computing resources allocated to inference requests, so as to achieve the purpose of optimizing the usage of computing resources of servers in the server cluster, has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The purpose of the embodiment of the present invention is to provide a resource scheduling method and device for large model reasoning requests, so as to reduce the possibility of lack or waste of computing resources allocated to reasoning requests by coordinating the graphics card resources and network bandwidth resources of the server cluster when computing resources are limited, thereby achieving the purpose of optimizing the use of computing resources of servers in the server cluster. The specific technical solution is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a resource scheduling method for a large model inference request, which is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, each of which is deployed with multiple large models, and the method includes:

[0009] Determine each target reasoning request to be scheduled for resources;

[0010] Determine the computing resource requirements corresponding to each target reasoning request; wherein the computing resource requirements corresponding to each target reasoning request are the amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource requirements corresponding to each target reasoning request include: graphics card parameters, video memory parameters, network bandwidth required by a candidate server for processing the target reasoning request, and model throughput of a large model for processing the target reasoning request when running in the candidate server;

[0011] Substituting the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request;

[0012] For each target reasoning request, the target reasoning request is allocated to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model;

[0013] The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is:

[0014] ;

[0015] Characterize the total number of target servers in the service cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes the second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand.

[0016] In a second aspect, an embodiment of the present invention provides a resource scheduling device for a large model inference request, which is applied to a scheduling server in a server cluster, wherein the server cluster further includes a plurality of target servers, each of which is deployed with a plurality of large models, and the device includes:

[0017] A request determination module, used to determine each target reasoning request to be scheduled for resources;

[0018] A demand determination module is used to determine the computing resource demand corresponding to each target reasoning request; wherein the computing resource demand corresponding to each target reasoning request is the resource amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource demand corresponding to each target reasoning request includes: graphics card parameters, video memory parameters, network bandwidth required by a candidate server for processing the target reasoning request, and model throughput of a large model for processing the target reasoning request when running in the candidate server;

[0019] A result determination module is used to substitute the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request;

[0020] An allocation module is used to allocate each target reasoning request to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model;

[0021] The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is:

[0022] ;

[0023] Characterize the total number of target servers in the service cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes the second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand.

[0024] In a third aspect, an embodiment of the present invention provides a server, comprising: a memory for storing computer programs; and a processor for implementing any of the resource scheduling methods for large model inference requests when executing the programs stored in the memory.

[0025] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the resource scheduling methods for large model inference requests.

[0026] An embodiment of the present invention also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described resource scheduling methods for large model inference requests.

[0027] Beneficial effects of the embodiments of the present invention:

[0028] As can be seen from the above, a resource scheduling method for large model inference requests provided by an embodiment of the present invention is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, and each target server is deployed with multiple large models. Before the scheduling server performs resource scheduling, first, a pre-trained resource prediction model is used to predict the amount of computing resources required when each inference request is responded to and processed, as the computing resource demand corresponding to the inference request, and the server performance ranking of each target server in the server cluster is pre-determined.

[0029] In this way, when performing resource scheduling, the scheduling server determines each target reasoning request to be scheduled, and the computing resource requirements corresponding to each target reasoning request. Therefore, the computing resource requirements corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the service cluster are substituted into the objective function to solve the allocation result of each target reasoning request, and based on the allocation result of each target reasoning request, the target server where the large model for responding to the target reasoning request is deployed is determined. In this way, for each target reasoning request, the target reasoning request is allocated to the target server represented by the corresponding allocation result, so that the target server that receives the target reasoning request determines the computing resource requirements corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource requirements, so as to process the target reasoning request through the deployed large model.

[0030] The computing resource requirements corresponding to each target reasoning request include: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model used to process the target reasoning request when it runs in the candidate server. In other words, when the scheduling server predicts the computing resource requirements, it takes into account the server's hardware graphics card resources and network bandwidth resources, that is, the graphics card parameters and video memory parameters of the server, as well as the network bandwidth of the server and the model throughput of the large model used to process the target reasoning request when it runs in the server.

[0031] Based on this, the solution provided by the embodiment of the present invention is applied. Based on the predicted computing resource demand corresponding to each target reasoning request and the server performance ranking of each target server in the server cluster, when computing resources are limited, by coordinating the graphics card resources and network bandwidth resources of the server cluster, after each target reasoning request is allocated, the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements in the actual reasoning scenario, so as to reduce the possibility of lack or waste of computing resources allocated to reasoning requests, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0032] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0034] Figure 1 A flowchart of a resource scheduling method for large model inference requests provided by an embodiment of the present invention;

[0035] Figure 2 A flow chart of a method for determining the server performance ranking of each target server in a server cluster provided by an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a judgment matrix provided by an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of a specific embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the structure of a resource scheduling device for large model reasoning requests provided by an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the protection scope of the present invention.

[0041] In order to facilitate the understanding of the present solution, the professional terms and related technologies involved in the embodiments of the present invention are first exemplified below.

[0042] A large model refers to a machine learning model with large-scale parameters and complex computing structures. These models are usually built from deep neural networks and have billions or even hundreds of billions of parameters. The purpose of designing a large model is to improve the expressiveness and predictive performance of the model and to be able to handle more complex tasks and data. By training massive data, a large model can learn complex patterns and features, has a stronger generalization ability, and can make accurate predictions on unseen data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Therefore, the large model involved in the embodiment of the present invention can be a large model for text processing, and accordingly, the request type of the inference request that can be processed by the large model can be a text recognition type; it can also be a large model for image processing, and accordingly, the request type of the inference request that can be processed by the large model can be an image recognition type; it can also be a large model for speech recognition, and accordingly, the type of inference request that can be processed by the large model can be a voiceprint recognition type. The present invention does not specifically limit the type of large model and the request type of inference request that can be processed.

[0043] The emergence of big model technology marks a new era of information processing and intelligent decision-making. These advanced models have unique advantages in understanding and generating semantic content with their powerful data processing capabilities and complex algorithm structures. Whether in text, audio, image, or video data, big models have demonstrated their powerful parsing and generation capabilities. As big models penetrate into all aspects of corporate operations, financial institutions are taking advantage of the advantages of big models to more efficiently sort out and analyze massive amounts of data, thereby improving the speed and accuracy of decision-making, achieving rich semantic understanding of big models, helping companies improve business performance, and gaining advantages in fierce market competition.

[0044] Normally, a large model uses the server on which the large model is deployed as the basic computing power facility. That is to say, the large model resources required to process each inference request (also called a large model inference request) are actually the computing resources provided by the server when the large model processes each inference request on the deployed server. However, the computing resources of the server currently face problems such as high resource costs. For example, the graphics card resources, network bandwidth and other resources of a high-performance server require a higher capital cost. For example, the graphics card resources, network bandwidth and other resources of a high-performance server require a higher capital cost.

[0045] For example, a computing power usage scheme guided by graphics card leasing is likely to cause a large amount of user data to go out of the domain, which not only increases the risk of user privacy leakage, but also may infringe on sensitive data property rights, which may cause user opposition and regulatory prohibition. However, if an enterprise uses a higher cost to build its own computing power center, then how to efficiently use resources and optimize the scheduling of computing resources within limited hardware graphics cards (also known as graphics card resources) and network bandwidth is a direct way to improve computing resource utilization, speed up large model inference speed, and optimize user experience.

[0046] In the related art, the cloud server performs graphics card resource allocation strategy planning based on the graphics card usage request information sent by the user terminal and the first total graphics card inventory information sent by the current first graphics card server.

[0047] In another related technology, a graphics card resource allocation strategy is planned based on the target resource value corresponding to the required resource information of each inference model under multi-model management. Thus, the graphics card resource value is pre-allocated according to the target resource value that meets the resource requirements of any model, and a model calculation group that can execute inference calculations of any model is generated, thereby realizing inference calculations of any model of graphics card resources and enabling the system to effectively utilize graphics card resources.

[0048] In another related technology, by building a distributed cluster consisting of multiple servers, sufficient graphics card resources are provided for the operation of large models. In this way, after receiving each submitted job, the graphics card resources required to process the job can be determined, and then the graphics card resources of the distributed cluster can be scheduled to provide the required graphics card resource requirements for the job.

[0049] In summary, in the execution process of the above-mentioned related technologies, when determining the graphics card resources, the real-time characteristic factors affecting the demand for graphics card resources, such as large model parameters, calling conditions, and user conditions, are not integrated, resulting in the determined graphics card resource demand being unpredictable.

[0050] In addition, in the case of limited computing resources, in order to ensure user experience, when performing resource scheduling, it is also necessary to ensure the coordinated scheduling of network resources and graphics card resources, that is, one computing resource cannot be considered alone. However, in the above-mentioned related technologies, only the scheduling of graphics card resources is considered, and the scheduling of other resources, such as the scheduling of network bandwidth resources, is not considered. As a result, when computing resources are limited, if a computing resource is insufficient or overflowed during the resource scheduling process, the server's computing resources are lacking or wasted.

[0051] That is to say, the resource scheduling solution for large model inference requests in the prior art has the following disadvantages:

[0052] (1) Currently, most resource scheduling schemes for large model inference requests are based on the demand application for computing resource requirements received by the management platform and perform corresponding computing resource allocation. However, the computing resource requirements of each inference request will be affected by real-time characteristic factors such as large model parameters, request type, and user conditions. Therefore, if the influence of factors such as large model parameters, request type, and user conditions is not considered, it may lead to insufficient computing resources for large models in actual inference scenarios, thus causing problems such as poor inference effect, incomprehensible output results, and poor user experience in the application of large models in business peak scenarios.

[0053] (2) The existing resource allocation and adjustment scheme for large model inference requests mainly relies on the current monitoring of resource consumption demand, that is, the corresponding computing resources are allocated to each inference request based on the current remaining computing resources. This scheme lacks a comprehensive prediction mechanism for graphics card resources and bandwidth resource consumption demand, which makes the resource allocation and adjustment for inference requests lagging, and ultimately leads to inadequate adjustment of graphics card resources or bandwidth resources, which reduces the user experience in the inference scenario, thereby affecting the user experience of large models.

[0054] (3) The existing resource allocation and adjustment strategies for large model inference requests usually allocate and adjust graphics card resources and bandwidth resources separately. This may lead to problems such as insufficient or overflow of certain adjustment resources during the resource allocation and adjustment process, thus causing a lack or waste of computing resources for large model inference requests.

[0055] In summary, how to establish a resource scheduling scheme that integrates graphics card resources and network bandwidth resources to optimize the automatic scheduling of resource allocation and adjustment for large model reasoning requests is the core problem that needs to be solved in view of the shortcomings of the above-mentioned related technologies. In other words, when performing resource scheduling, comprehensively consider influencing factors such as large model parameters, request type, and user conditions, build a prediction mechanism for resources such as comprehensive graphics card resources and network bandwidth, predict graphics card resources and network bandwidth resources when providing reasoning services for large models, and, based on the prediction results obtained for graphics card resources and network bandwidth resources, implement effective resource allocation and adjustment strategies for large model reasoning requests to improve resource utilization efficiency, ensure the stability of related reasoning services for large models, and improve user experience. This is a technical problem that urgently needs to be solved.

[0056] In short, the technical problem to be solved by the present invention is: how to reduce the possibility of shortage or waste of computing resources allocated to inference requests by collaboratively scheduling the graphics card resources and network bandwidth resources of the server cluster when computing resources are limited, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0057] Based on this, in order to solve the above technical problems, an embodiment of the present invention provides a resource scheduling method and device for large model inference requests.

[0058] Among them, the method is applied to a scheduling server in a server cluster. In a specific application, the server can be an architecture server, a cabinet server or a blade server, etc. The present invention does not limit the specific form of the server.

[0059] Furthermore, the method is applicable to various application scenarios for scheduling resources for large model inference requests. For example, resources for inference requests for processing images, resources for inference requests for processing video data, and resources for inference requests for processing speech recognition are scheduled. The embodiment of the present invention does not specifically limit the application scenarios of the method.

[0060] An embodiment of the present invention provides a resource scheduling method for a large model inference request, which is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, each of which is deployed with multiple large models, and the method includes:

[0061] Determine each target reasoning request to be scheduled for resources;

[0062] Determine the computing resource requirements corresponding to each target reasoning request; wherein the computing resource requirements corresponding to each target reasoning request are the amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource requirements corresponding to each target reasoning request include: graphics card parameters, video memory parameters, network bandwidth required by a candidate server for processing the target reasoning request, and model throughput of a large model for processing the target reasoning request when running in the candidate server;

[0063] Substituting the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request;

[0064] For each target reasoning request, the target reasoning request is allocated to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model;

[0065] The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is:

[0066] ;

[0067] Characterize the total number of target servers in the service cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes the second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand.

[0068] As can be seen from the above, a resource scheduling method for large model inference requests provided by an embodiment of the present invention is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, and each target server is deployed with multiple large models. Before the scheduling server performs resource scheduling, first, a pre-trained resource prediction model is used to predict the amount of computing resources required when each inference request is responded to and processed, as the computing resource demand corresponding to the inference request, and the server performance ranking of each target server in the server cluster is pre-determined.

[0069] In this way, when performing resource scheduling, the scheduling server determines each target reasoning request to be scheduled, and the computing resource requirements corresponding to each target reasoning request. Therefore, the computing resource requirements corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the service cluster are substituted into the objective function to solve the allocation result of each target reasoning request, and based on the allocation result of each target reasoning request, the target server where the large model for responding to the target reasoning request is deployed is determined. In this way, for each target reasoning request, the target reasoning request is allocated to the target server represented by the corresponding allocation result, so that the target server receiving the target reasoning request determines the computing resource requirements corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource requirements, so as to process the target reasoning request through the deployed large model.

[0070] The computing resource requirements corresponding to each target reasoning request include: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model used to process the target reasoning request when it runs in the candidate server. In other words, when the scheduling server predicts the computing resource requirements, it takes into account the server's hardware graphics card resources and network bandwidth resources, that is, the graphics card parameters and video memory parameters of the server, as well as the network bandwidth of the server and the model throughput of the large model used to process the target reasoning request when it runs in the server.

[0071] Based on this, the solution provided by the embodiment of the present invention is applied. Based on the predicted computing resource demand corresponding to each target reasoning request and the server performance ranking of each target server in the server cluster, when computing resources are limited, by coordinating the graphics card resources and network bandwidth resources of the server cluster, after each target reasoning request is allocated, the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements in the actual reasoning scenario, so as to reduce the possibility of lack or waste of computing resources allocated to reasoning requests, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0072] In conjunction with the accompanying drawings, a resource scheduling method for large model inference requests provided by an embodiment of the present invention is specifically introduced below.

[0073] like Figure 1 As shown, a resource scheduling method for a large model inference request provided by an embodiment of the present invention is applied to a scheduling server in a service cluster, the server cluster also includes multiple target servers, each target server is deployed with multiple large models, and the method may include the following steps:

[0074] S101: Determine each target inference request to be scheduled for resources.

[0075] In the present invention, when performing resource scheduling, the scheduling server first determines each target reasoning request to be scheduled, and then executes a resource scheduling method for large model reasoning requests provided by an embodiment of the present invention for each target reasoning request.

[0076] Optionally, the scheduling server may receive each request according to a predetermined receiving cycle, and determine all requests received in the receiving cycle as target inference requests.

[0077] It should be noted that a large model can support at least one type of inference request. For example, a large model used for text processing can support text recognition type requests; a large model used for image processing can support image recognition type requests; a large model used for audio processing can support audio noise reduction type requests. Therefore, in order to improve the response efficiency of the entire server cluster to inference requests, multiple large models can be deployed in each target server in the server cluster. In this way, each target server can use each large model deployed by itself to process inference requests of the supported request types.

[0078] S102: Determine the computing resource requirements corresponding to each target inference request.

[0079] Among them, the computing resource demand corresponding to each target reasoning request is the amount of computing resources required to respond to and process the target reasoning request, which is predicted by a pre-trained resource prediction model; the computing resource demand corresponding to each target reasoning request includes: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model used to process the target reasoning request when it runs in the candidate server.

[0080] In the present invention, a resource prediction model that is pre-trained and used to predict the computing resource requirements of each inference request can be deployed in the scheduling server. In this way, before performing resource scheduling, the scheduling server can use the above resource prediction model to predict the computing resource requirements required for each inference request to be responded to and processed.

[0081] It should be understood that the above-mentioned resource prediction model predicts the computing resource demand corresponding to the request type of each inference request received by the scheduling server within this receiving cycle. That is to say, the general process of a resource scheduling method for large model inference requests provided by an embodiment of the present invention is described from the perspective of the scheduling server as follows: after receiving each inference request within a receiving cycle, the scheduling server uses the resource prediction model to predict the computing resource demand corresponding to each inference request within the receiving cycle, and then determines the allocation result corresponding to each inference request based on the predicted computing resource demand corresponding to each inference request.

[0082] For ease of reading, the specific implementation method of the training method of the resource prediction model will be described in conjunction with the following embodiments and will not be repeated here.

[0083] In this way, during the actual reasoning process, after receiving each target reasoning request, the scheduling server can determine the computing resource demand corresponding to the target reasoning request based on the request type of each target reasoning request.

[0084] Among them, the computing resource demand corresponding to each target inference request predicted by the scheduling server using the resource prediction model includes: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target inference request, and the model throughput of the large model used to process the target inference request when running in the candidate server.

[0085] That is to say, when the scheduling server predicts the demand for computing resources, it comprehensively considers the server's hardware graphics card resources and network bandwidth resources. In this way, when resources are subsequently allocated and scheduled, it can reduce the possibility of shortage or waste of computing resources allocated to inference requests by coordinating graphics card resources and network bandwidth resources, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0086] Optionally, in one implementation, graphics card parameters, video memory parameters, network bandwidth, and model throughput of the large model when running on each target server are different.

[0087] In this implementation, the graphics card parameters, video memory parameters, network bandwidth of each target server and the model throughput of the large model when running on the server are different, so that the server cluster composed of the target servers contains a variety of different types of graphics card resources and network bandwidth resources, providing a variety of graphics card resources and network bandwidth resources for resource scheduling of the scheduling server.

[0088] S103: Substitute the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request;

[0089] S104: For each target reasoning request, the target reasoning request is allocated to the target server represented by the allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model.

[0090] The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the above objective function is:

[0091] ;

[0092] Represents the total number of target servers in the service cluster; Represents the total number of inference requests for each target; Characterization The server performance ranking of the target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value is 1, it indicates The target inference request is assigned to target server for processing. When the value is 0, it indicates Target inference requests were not assigned to target server for processing; Indicates that when the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes the second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of resource utilization requirements; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of resource utilization requirements.

[0093] In the present invention, in combination with the actual resource scheduling process, the situation of prioritizing resource balance or prioritizing server performance, the user can pre-set the size relationship between the first weight and the second weight according to the resource utilization requirements in the actual reasoning scenario. Thus, based on the size relationship between the two weights, the allocation target of the resource utilization requirements when allocating resources for each reasoning request is determined, thereby achieving efficient utilization of the computing resources of each server in the server cluster. Specifically: when the first weight is greater than the second weight, it represents the priority of resource allocation balance as the allocation target of resource utilization requirements. On the contrary, when the first weight is less than the second weight, it represents the priority of server performance as the allocation target of resource utilization requirements.

[0094] For example, when there are many inference requests, in order to ensure the stability of the large model operation and process as many inference requests as possible, the user can set the first weight to be smaller than the second weight so that when allocating resources, server performance is prioritized as the allocation target for resource utilization needs.

[0095] Before performing resource scheduling, the scheduling server can predetermine the server performance ranking of each target server in the server cluster, thereby substituting the computing resource demand corresponding to each target inference request, the current remaining resource amount of each target server, and the server performance ranking of each target server into the pre-designed objective function, thereby selecting the maximum value from each variance about the resource proportion as the value multiplied by the first weight; wherein each variance value about the resource proportion is the maximum value in the objective function. ; After determining the target sum of the product of the server performance of each target server in the server cluster and the current remaining resource ratio of each target server, determine the mean value corresponding to each target sum value, and use the target mean value as the value multiplied by the second weight; wherein the mean value corresponding to each target sum value is the value in the objective function , the above target mean refers to the value that can make , the mean of the maximum and values ​​can be achieved.

[0096] Thus, by substituting the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the above objective function, the target server can be determined by The value is 1 or 0, which determines whether The target reasoning request is assigned to target server for processing.

[0097] Specifically, in When the value of is 1, it is determined that The target inference request is assigned to The target server will process it. The target server is identified as the one that responds to The target server where the large model for target reasoning requests is deployed; When the value of is 0, determine the The target inference request cannot be assigned to target server to process. Understandably, When the value of is 0, value, greater than When the value of is 1, Therefore, in order to ensure that resource scheduling is performed based on the obtained allocation results so that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, the computing resource requirements corresponding to each target inference request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the server cluster are substituted into the above objective function. When the value is minimum, The letter with the smallest value The target reasoning request represented is assigned to the letter In the target server represented by The target server characterizes the response letter The target server on which the large model of the represented target reasoning request is deployed is used to determine the allocation result of each target reasoning request.

[0098] The specific implementation method of determining the server performance ranking of each target server in the server cluster is as described in steps S201 to S206 below, which will not be repeated here.

[0099] Afterwards, after determining the target server to which each target reasoning request needs to be assigned and represented by the allocation result of the target reasoning request, the scheduling server can allocate the target reasoning request to the corresponding target server, so that the target server that receives the target reasoning request can perform resource scheduling for the deployed large model according to the determined computing resource demand after determining the computing resource demand corresponding to the target reasoning request, thereby processing the target reasoning request through the large model deployed on the target server.

[0100] Considering that the overall solution process of the above objective function is too complicated, based on this, optionally, a predetermined local solution method is used to solve the above objective function to obtain the allocation result of each target reasoning request, so as to improve the solution efficiency of the objective function, thereby improving the response efficiency of the large model to the target reasoning request. Among them, the above predetermined local solution method can be an integer programming solution method, a linear solution method, etc., which is not specifically limited in the embodiment of the present invention.

[0101] In addition, each time an inference request is processed, the amount of computing resources occupied by the inference request will be released. At this time, optionally, after the scheduling server determines the allocation result of each target inference request and performs resource scheduling on the target inference request based on the allocation result, it detects whether there is resource release in the server cluster. If so, the current remaining resources of each target server are updated based on the released resources.

[0102] As can be seen from the above, a resource scheduling method for large model inference requests provided by an embodiment of the present invention is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, and each target server is deployed with multiple large models. Before the scheduling server performs resource scheduling, first, a pre-trained resource prediction model is used to predict the amount of computing resources required when each inference request is responded to and processed, as the computing resource demand corresponding to the inference request, and the server performance ranking of each target server in the server cluster is pre-determined.

[0103] In this way, when performing resource scheduling, the scheduling server determines each target reasoning request to be scheduled, and the computing resource requirements corresponding to each target reasoning request. Therefore, the computing resource requirements corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the service cluster are substituted into the objective function to solve the allocation result of each target reasoning request, and based on the allocation result of each target reasoning request, the target server where the large model for responding to the target reasoning request is deployed is determined. In this way, for each target reasoning request, the target reasoning request is allocated to the target server represented by the corresponding allocation result, so that the target server that receives the target reasoning request determines the computing resource requirements corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource requirements, so as to process the target reasoning request through the deployed large model.

[0104] The computing resource requirements corresponding to each target reasoning request include: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model used to process the target reasoning request when it runs in the candidate server. In other words, when the scheduling server predicts the computing resource requirements, it takes into account the server's hardware graphics card resources and network bandwidth resources, that is, the graphics card parameters and video memory parameters of the server, as well as the network bandwidth of the server and the model throughput of the large model used to process the target reasoning request when it runs in the server.

[0105] Based on this, the solution provided by the embodiment of the present invention is applied. Based on the predicted computing resource demand corresponding to each target reasoning request and the server performance ranking of each target server in the server cluster, when computing resources are limited, by coordinating the graphics card resources and network bandwidth resources of the server cluster, after each target reasoning request is allocated, the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements in the actual reasoning scenario, so as to reduce the possibility of lack or waste of computing resources allocated to reasoning requests, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0106] Optionally, in one embodiment, as Figure 2 As shown, in a resource scheduling method for a large model inference request provided by an embodiment of the present invention, a method for determining the server performance ranking of each target server in a server cluster may include the following steps:

[0107] S201: for each target server, performing predetermined data standardization processing on each designated performance indicator of the target server to obtain standardized data corresponding to each designated performance indicator of the target server, and determining weight parameters corresponding to each designated performance indicator of the target server based on the standardized data corresponding to each designated performance indicator;

[0108] S202: generating a target matrix for server performance of the service cluster based on weight parameters and standardized data corresponding to each designated performance indicator of each target server;

[0109] Wherein, each row of matrix elements in the target matrix is ​​used to characterize the server performance of a target server;

[0110] S203: Determine the maximum value of each column in the target matrix, and combine them to obtain a positive ideal solution for the target matrix;

[0111] S204: Determine the minimum value of each column in the target matrix, and combine to obtain a negative ideal solution of the target matrix;

[0112] S205: for each target server, determining a first distance between each designated performance indicator of the target server and the positive ideal solution, and a second distance between each designated performance indicator of the target server and the negative ideal solution, and determining a relative closeness corresponding to the target server based on the obtained first distance and second distance;

[0113] The relative closeness corresponding to the target server is the relative closeness of each designated performance indicator of the target server to the positive ideal solution, or the relative closeness to the negative ideal solution;

[0114] S206: Sort the target servers based on the relative closeness corresponding to the target servers to obtain a server performance ranking of each target server in the server cluster.

[0115] In this embodiment, before scheduling resources for each target reasoning request, the scheduling server determines the server performance ranking of each target server in the server cluster in advance for each target server in the service cluster. Specifically:

[0116] When considering that different servers are equipped with different graphics cards or processors, their computing rates and transmission speeds are often different, and the storage, network, networking mode, etc. of each server will affect the computing power of the server, that is, the server performance of the server. Therefore, to comprehensively consider the computing power index of the server, it can be measured from the perspective of the server's transaction processing. In other words, the transaction processing rate of each target server is used as a specified performance index of the target server, and the hardware performance of the processor carried by the target server itself, that is, the floating-point computing performance of the processor, the video memory capacity of the processor, and the bandwidth of the processor, are also used as the specified performance indicators of the target server. Thus, optionally, the multiple specified performance indicators of each target server include: the transaction processing rate of the target server, the floating-point computing performance of the processor carried by the target server, the video memory capacity of the processor, and the bandwidth of the processor. Among them, the processor carried by the above-mentioned target server can be a GPU (Graphics Processing Unit, graphics processor), which is not specifically limited in this embodiment of the present invention.

[0117] In this way, for each target server, by determining the model of the processor carried by the target server, the floating-point computing performance, the video memory capacity and the bandwidth of the processor carried by the target server are determined. And according to the traditional test standard, based on the transaction volume processed by the server per minute, the transaction processing rate corresponding to the target server is determined; wherein, the above-mentioned traditional test standard is the TPC-C (Transaction Processing Performance Council) standard, which is an industry standard benchmark test project designed to measure the performance and scalability of online transaction processing (OLTP) systems. This embodiment of the present invention does not make specific limitations.

[0118] Considering that different designated performance indicators have different dimensions, in order to eliminate the impact of dimensions, each designated performance indicator of the target server may be subjected to predetermined data standardization processing to obtain standardized data corresponding to each designated performance indicator of the target server.

[0119] Optionally, for each target server, based on each designated performance indicator of the target server, a predetermined first formula is used to perform data standardization processing on the designated performance indicator to obtain standardized data corresponding to the designated performance indicator; the first formula includes:

[0120] ;

[0121] in, Characterizes the service cluster The target server The standardized data corresponding to the specified performance indicator, Characterizes the service cluster The target server The raw data of the specified performance indicators; Represents the first The maximum raw data in the specified performance indicator; Represents the first The item specifies the minimum raw data in the performance metric.

[0122] Afterwards, based on the standardized data corresponding to each designated performance indicator, weight parameters corresponding to each designated performance indicator of the target server are determined.

[0123] For example, the weight parameters corresponding to different specified performance indicators are determined using the hierarchical analysis method. Figure 3The comparison judgment matrix shown in the figure shows that tpmC in the matrix represents the transaction processing rate of the target server, TF represents the floating-point computing performance of the processor on the target server, X represents the video memory capacity of the processor, and Y represents the bandwidth of the processor. Among them, the value corresponding to tpmC is 1, the value corresponding to TF is q, the value corresponding to X is w, and the data corresponding to Y is e. Figure 3 The comparison judgment matrix shown in FIG. 1 compares each two specified performance indicators in pairs.

[0124] It should be emphasized that the specific values ​​of the elements q, w and e in the above comparison judgment matrix are determined by the actual scenario. For example, a target server needs to take into account the reasoning requests received by the large model processing, and needs to meet user needs by operating the database, that is, the reasoning request needs to query data in the database and feedback to the user. At this time, the specific values ​​of the above elements q, w and e should be small; on the contrary, a target server only considers the reasoning requests received by the large model processing, that is, the reasoning request only needs to feedback the reasoning results to the user, and does not need to query the data in the database and then feedback to the user. At this time, the specific values ​​of the above elements q, w and e should be large. Usually, in the absence of quantitative data, this part often needs to give relative importance based on expert experience or neural networks. Therefore, the above elements q, w and e are hyperparameters.

[0125] In getting Figure 3 After obtaining the comparison judgment matrix shown, the comparison judgment matrix can be solved to obtain the maximum eigenvalue of the comparison judgment matrix. After determining the eigenvector corresponding to the maximum eigenvalue, the consistency index corresponding to the comparison judgment matrix is ​​calculated. Then, based on the consistency index and the consistency index determined by the matrix order of the comparison judgment matrix, the consistency ratio of the comparison judgment matrix is ​​calculated.

[0126] In this way, when the consistency ratio is greater than a predetermined value, it means that the set comparison judgment matrix cannot meet the consistency requirements. At this time, it is necessary to readjust the specific values ​​of the elements q, w and e in the comparison judgment matrix, and return to the step of solving the comparison judgment matrix after the elements are updated. When the consistency ratio is less than a predetermined value, it means that the set comparison judgment matrix meets the consistency requirements. At this time, the eigenvector corresponding to the maximum eigenvalue is normalized to obtain a row vector feature including the four specified performance indicators of tpmC, TF, X and Y, wherein each column element in the row vector feature is, in turn, the weight parameter corresponding to tpmC, the weight parameter corresponding to TF, the weight parameter corresponding to X and the weight parameter corresponding to Y. The above is only an exemplary order of the arrangement order of each column element in the row vector feature. The present invention does not limit the arrangement order between the elements in the row vector feature.

[0127] It should be emphasized that the above-mentioned method of using the hierarchical analysis method to determine the weight parameters corresponding to different specified performance indicators is only an exemplary specific implementation method provided by the present invention. The present invention does not specifically limit the specific implementation method for determining the weight parameters corresponding to different specified performance indicators.

[0128] Then, based on the weight parameters and standardized data corresponding to each designated performance indicator of each target server, a target matrix about the server performance of the server cluster is generated. In the target matrix, each row of matrix elements represents the server performance of a target server; each column of matrix elements represents a designated performance indicator of each target server in the service cluster. The value of each matrix element in each row of matrix elements is the product value of the designated performance indicator corresponding to a target server, its corresponding weight parameter and standardized data.

[0129] Considering that the specified performance indicators of each target server are all efficiency indicators, that is, the larger the specified performance indicator is, the stronger the server performance of the target server is. Therefore, for each target server, the closer the combination of all its specified performance indicators is to the maximum value of each column in the matrix, the better the server performance of the target server is; on the contrary, the closer the combination of all its specified performance indicators is to the minimum value in the matrix, the worse the server performance of the target server is.

[0130] Therefore, after generating the target matrix of the server performance of the server cluster, the maximum value of each column in the target matrix is ​​determined, and the positive ideal solution of the target matrix is ​​obtained by combination; and the minimum value of each column in the target matrix is ​​determined, and the negative ideal solution of the target matrix is ​​obtained by combination. Then, for each target server, the first distance between each specified performance indicator of the target server and the positive ideal solution and the second distance between each specified performance indicator of the target server and the negative ideal solution are determined, so that the relative closeness corresponding to the target server is determined based on the obtained first distance and second distance.

[0131] It should be clear that the purpose of determining the relative closeness is to rank the server performance of each target server in the server cluster based on the relative closeness. Therefore, the relative closeness corresponding to the target server can be understood as the relative closeness of each specified performance indicator of the target server to the positive ideal solution, or it can be understood as the relative closeness of each specified performance indicator of the target server to the negative ideal solution, which is reasonable.

[0132] Therefore, after determining the relative closeness corresponding to each target server, each target server can be ranked based on the relative closeness to obtain the server performance ranking of each target server in the server cluster.

[0133] Optionally, in an embodiment, the above step S101, determining each target reasoning request to be scheduled for resources, may include the following steps:

[0134] Step A1: for each inference request among the multiple inference requests that have been received but not processed, determine the computing resource requirement corresponding to the inference request, and use the computing resource requirement corresponding to the inference request and the predetermined model weight of the large model used to process the inference request as input content, and input them into a pre-trained request priority determination model to obtain the priority of the inference request;

[0135] Step A2: determining each target inference request to be resource scheduled from the multiple inference requests based on the priority corresponding to each inference request in the multiple inference requests;

[0136] The method for determining the computing resource requirement corresponding to the inference request may include the following steps:

[0137] Step B1: for each large model, obtain the resource usage required by the large model when processing historical reasoning requests of the target type, and obtain the resource usage of the large model when processing each historical reasoning request;

[0138] The historical reasoning request of the target type is a request that has completed reasoning and is of the same category as the reasoning request;

[0139] Step B2: Input the resource usage of each large model when processing each historical reasoning request into the pre-trained resource prediction model to obtain the computing resource demand corresponding to the reasoning request.

[0140] In this embodiment, for each large model, the resource usage required by the large model when processing historical reasoning requests of the target type is obtained in advance, and the resource usage of the large model when processing each historical reasoning request is obtained.

[0141] Optionally, the resource usage may include the large model parameters, request types and user conditions of the large model. The large model parameters include the model parameter quantity, the number of model layers and the output dimension of the attention network; the request type includes the unit length of the model input, the application scenario, and the computing power utilization required by the server running the large model; the user condition is the number of requests that the large model can handle per unit time, including the average concurrency of the large model when processing requests and the peak reservation multiple of computing resources.

[0142] After obtaining the resource usage of each large model when processing each historical reasoning request, the resource usage can be input into the pre-trained resource prediction model to obtain the computing resource demand corresponding to each reasoning request of the same category as the historical reasoning request.

[0143] In addition, considering that different types of inference requests have different importance levels of the corresponding large models that can process the inference requests of this type, the predetermined model weight corresponding to each large model can be pre-set. The larger the weight of the large model, the higher its importance in the actual inference scenario, and thus the priority of the inference request that it can process is higher than the priority of the inference request that other large models can process. For example, the importance level of large model l for video processing is higher than that of large model o for text processing. Therefore, the predetermined model weight of large model l can be set higher than the predetermined model weight of large model o. Accordingly, the priority of the inference request that large model l can process is higher than the priority of the inference request that large model o can process.

[0144] In this way, after determining the computing resource demand corresponding to each inference request among the multiple unprocessed inference requests that have been received, the computing resource demand corresponding to the inference request and the predetermined model weight of the large model used to process the inference request are used as input content and input into the pre-trained request priority determination model to obtain the priority of the inference request. Thus, based on the priority corresponding to each inference request, each target inference request to be scheduled for resources is determined from the multiple inference requests.

[0145] Optionally, the training method of the resource prediction model includes the following steps:

[0146] For each large model, the resource occupancy information required by the large model for processing historical reasoning requests of a certain request type collected in a predetermined time period is obtained as a set of training samples for training a resource prediction model. Then, by manually setting the number of hidden layers and neurons contained in the first initial model, the first initial model outputs the computing resource requirements for each reasoning request of the same request type when the input content is the above resource occupancy information. After the setting is successful, the sets of training samples collected in the predetermined time period are input into the first initial model after the setting is successful, and the resource prediction model is obtained through training.

[0147] The first initial model is a three-layer BP neural network model (Back Propagation Neural Network Model), which is not specifically limited in the present invention.

[0148] Optionally, the request prioritization model is trained in a manner including:

[0149] The computing resource demand corresponding to each inference request predicted by the resource prediction model and the predetermined model weight of the large model used to process the inference request are used as a group of training samples. Then, each group of training samples is normalized, and the normalized sample data are used as the training group and the test group respectively. After that, the second initial model is trained using the sample data in the training group, and the trained second initial model is adjusted using the test group, that is, the structural parameters such as the number of hidden layers, the number of nodes, the activation function and the weight function of the second initial model are adjusted until the loss value of the trained model is less than the predetermined loss value, indicating that the request priority determination model training is completed.

[0150] The second initial model may be a DBN (Deep Belief Networks, also called a deep belief network), which is not specifically limited in the embodiment of the present invention.

[0151] In this embodiment, the resource prediction model and request priority determination model obtained through pre-training are used to determine the request priorities corresponding to multiple received but unprocessed inference requests. The so-called request priority represents the importance of the inference request among multiple inference requests. In this way, when performing resource scheduling for inference requests, inference requests with high importance and high request priority can be given priority based on the request priority.

[0152] Optionally, in one embodiment, the received unprocessed multiple inference requests are inference requests received within at least one receiving cycle; in the above step A2, based on the priority corresponding to each inference request in the multiple inference requests, before determining each target inference request to be resource scheduled from the multiple inference requests, a resource scheduling method for large model inference requests provided by an embodiment of the present invention may further include the following steps:

[0153] Step C1: determining an inference request that meets the target selection condition among multiple inference requests;

[0154] The target selection condition includes that the target does not belong to the inference request received in the most recent receiving cycle;

[0155] Step C2: Raise the priority level corresponding to the determined inference request.

[0156] In this embodiment, considering that a large number of inference requests are received within a receiving cycle, there are unprocessed and remaining inference requests within the receiving cycle, that is, the multiple unprocessed inference requests received are inference requests received within at least one receiving cycle.

[0157] In order not to delay the processing efficiency of the reasoning request, based on the priority corresponding to each reasoning request among the multiple reasoning requests, before determining the target reasoning requests to be resource scheduled from the multiple reasoning requests, the reasoning requests among the multiple reasoning requests that do not belong to the most recent receiving cycle are determined, and the priority corresponding to the determined reasoning request is increased.

[0158] In this embodiment, the priority of the remaining inference requests in other receiving cycles before the current receiving cycle is adjusted to increase the importance of the inference request. Therefore, when determining the target inference request and performing resource scheduling, the remaining inference requests in other receiving cycles can be given priority to reduce the possibility of untimely feedback on the remaining inference requests.

[0159] Optionally, in one embodiment, a resource scheduling method for a large model inference request provided by an embodiment of the present invention may further include the following steps:

[0160] Step D: setting a selection condition for a target server for processing each target reasoning request according to the computing resource demand; wherein the selection condition is used to limit each target reasoning request to correspond to a target server for processing the target reasoning request;

[0161] Correspondingly, the above step S103, which substitutes the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request, may include the following steps:

[0162] Step E: Substitute the computing resource demand corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function, and combine the selection conditions to determine the allocation result of each target reasoning request.

[0163] In this embodiment, under normal circumstances, a target reasoning request can only be assigned to a large model deployed by a target server for processing. Therefore, according to the computing resource demand corresponding to each target reasoning request, selection conditions for the target server used to process each target reasoning request are set to limit each target reasoning request to correspond to one target server used to process the target reasoning request.

[0164] In this way, in the process of substituting the computing resource demand corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to determine the allocation result of each target reasoning request, by combining the above-mentioned selection conditions, the number of target servers corresponding to each target reasoning request is limited, thereby reducing the possibility of multiple allocations of the same target reasoning request, and then reducing the possibility of lack or waste of computing resources allocated to reasoning requests, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0165] The following is an exemplary introduction to a resource scheduling method for large model inference requests provided by the present invention in conjunction with a specific embodiment. Figure 4 The figure is a flow chart of a specific embodiment provided by the embodiment of the present invention.

[0166] Among them, this specific embodiment is divided into two parts in resource scheduling for large model inference requests:

[0167] The first part includes determining the model parameters, call status and user status of the large model. Among them, the model parameters include the model parameter quantity, the number of model layers and the output dimension of the attention network; the call status includes: token length, supported task scenarios (text, images, etc.) and computing power utilization; the user status includes the average concurrency and peak reservation multiples; among them, the above call status is used to reflect the request type that the model can handle, that is, the above call status is the request type in the present invention.

[0168] First, in the first part above, in view of the existing problems in the above-mentioned prior art, when determining the computing resource requirements, factors such as large model parameters, request type, and user conditions are not considered, resulting in a lag problem in resource allocation and adjustment for large model reasoning requests, thereby leading to different computing resources in large model reasoning scenarios, resulting in poor reasoning effects, unsatisfactory output results, and poor user experience for large models in business peak scenarios. This specific embodiment provides a method for predicting the computing resource requirements for large model reasoning requests. By considering factors such as large model parameters, request type, and user conditions, the graphics card resources and network bandwidth resources for large model reasoning requests are predicted to improve the accuracy of the predicted computing resource requirements.

[0169] The following is a detailed introduction to a prediction method provided:

[0170] Considering the difficulty in counting the computing resources provided by the server required to process each inference request for a large model, the reasons are:

[0171] (1) The resource requirements of large models are affected by multiple factors, and there are complex nonlinear relationships between these factors;

[0172] (2) Large models usually have billions or even hundreds of billions of parameters, and the relationship between the parameter size and the amount of computing resources required is nonlinear;

[0173] (3) The structure and details of large models will affect the usage of computing resources;

[0174] (4) Large models may dynamically adjust the use of computing resources as needed during operation;

[0175] (5) Large models may be idle and their use of computing resources is unstable.

[0176] Based on the above exemplary reasons, the process of predicting the demand for computing resources is actually a prediction of the computing resources required for the above-mentioned characteristics of the large model, thereby reducing the problem of inaccurate resource scheduling for inference requests due to inaccurate analysis of the computing resources required by the large model itself.

[0177] Therefore, based on the model parameters of each large model, the request types of inference requests that can be processed, and the user situation, the demand for four resources, namely, graphics card G, video memory V, bandwidth B, and throughput T, when the large model processes each inference request is predicted. Among them, graphics card G is the graphics card parameter of the server equipped with the large model, video memory V is the video memory parameter of the server equipped with the large model, bandwidth B is the network bandwidth of the server equipped with the large model, and throughput T is the model throughput when the large model is running in the server. Specifically:

[0178] First, the real resource occupancy information of the large model during operation is collected periodically as a training sample for the required resource prediction. The resource occupancy information is the resource occupancy amount in the present invention. Considering that a three-layer BP neural network can sufficiently approximate any nonlinear function, in this specific embodiment, a three-layer BP neural network is used for resource prediction.

[0179] Then, the resource occupancy information is used as the input of the BP neural network, and the graphics card G, video memory V, bandwidth B and throughput T are used as the output of the BP neural network. For example, the following formula is used to represent the input of the BP neural network:

[0180] ;

[0181] is the model parameter, is the number of model layers, is the output dimension of the attention model, is the token length, To support the mission scenarios, is the computing power utilization rate, To allow maximum concurrency, Reserve multiples for peak values.

[0182] Among them, the supported task scenarios are the application scenarios of the large model in the embodiment of the present invention; the maximum allowed concurrency is the average concurrency when the large model processes requests in the embodiment of the present invention; the peak reservation multiple is the peak reservation multiple for computing resources in the embodiment of the present invention.

[0183] Finally, the number of hidden layers and the number of neurons in the three-layer BP neural network are determined, and the three-layer BP neural network is trained using the collected training samples to obtain the resource prediction model in the embodiment of the present invention.

[0184] After receiving each inference request, the resource prediction model can be used to predict the graphics card parameter requirements for each inference request, as well as the network situation requirements for each inference request. Specifically, the graphics card G, video memory V, bandwidth B and throughput T corresponding to each inference request are predicted; wherein, the graphics card G is used to characterize the computing core of the graphics card, that is, the parameters of the graphics processor integrated in the graphics card (i.e., the graphics card parameters of the server in the embodiment of the present invention); the video memory V is used to characterize the graphics card MEM (memory) parameters (i.e., the video memory parameters of the server in the embodiment of the present invention); the throughput T is used to characterize the network throughput (i.e., the model throughput in the embodiment of the present invention); and the bandwidth B is used to characterize the network bandwidth.

[0185] It should be noted that the specific implementation method of training the three-layer BP neural network is similar to the specific implementation method of training the resource prediction model in the embodiment of the present invention, and will not be repeated here.

[0186] Usually, a server can load multiple large models of different types, and different types of large models have different importance in actual reasoning scenarios. Therefore, for large models with higher importance, the importance of the reasoning requests they can process is also higher among other reasoning requests, that is, the reasoning requests that can be processed by large models with higher importance have higher request priority. In this way, when there are multiple reasoning requests, the reasoning requests with higher request priority will be used to determine the target reasoning requests to be scheduled, and resource scheduling and processing will be prioritized for each target reasoning request.

[0187] Therefore, this specific embodiment provides a large model weight evaluation model obtained by deep belief network training, that is, the request priority determination model provided by the embodiment of the present invention. Using the large model weight evaluation model, according to the computing resources required by each large model and the importance of the large model, the request priority of the inference request that can be processed by the large model is comprehensively evaluated to determine the efficient use of computing resources on the basis of achieving the normal operation of more important large models. Specifically:

[0188] The computing resource requirements corresponding to the graphics card, video memory, bandwidth and throughput respectively predicted by the above resource prediction model, as well as the predetermined model weights determined in advance according to the importance of the large model, are used as a group of training samples to establish a data set for training a large model weight evaluation model, and the Z-Score (Z score, also known as the standard score) method is used to normalize each group of training samples in the above data set. Then, the normalized data set is divided into a training group and a test group.

[0189] Afterwards, the computing resource requirements corresponding to the graphics card, video memory, bandwidth and throughput, as well as the predetermined model weights of the large model, are used as network input parameters of the deep belief network; and the deep belief network is trained using the above training group, and then the trained deep belief network is adjusted using the test group, that is, the structural parameters such as the number of hidden layers, the number of nodes, the activation function and the weight function of the deep belief network are adjusted until the loss value of the trained model is less than the predetermined loss value, indicating that the training of the large model weight evaluation model is completed.

[0190] It should be noted that the specific implementation method of training the deep belief network is similar to the specific implementation method of training the request priority determination model in the embodiment of the present invention, and will not be repeated here.

[0191] After the first part predicts the four computing resources of graphics card G, video memory V, bandwidth B and throughput T, that is, the graphics card parameter demand prediction results and the network demand prediction results, we can enter the second part of resource scheduling based on the above two prediction results, that is, to effectively allocate and adjust the computing resources for large model inference requests, realize the coordinated allocation and automatic adjustment of graphics card resources and network bandwidth resources, efficiently utilize computing resources, and effectively maximize the resource value of computing resources, thereby improving the utilization efficiency of computing resources, ensuring the stability of large model related inference services, and further improving user experience.

[0192] Considering that in the application stage, the large model needs to be deployed on the server to process the reasoning request from the user. In this process, two steps, deployment and reasoning, are included. Among them, the deployment process is actually the process of model loading. When multiple large models are loaded on the server cluster, each large model is loaded once on other servers (target servers) in the server cluster except the server (scheduling server) for resource scheduling. In the process of loading the large model, more computing resources need to be occupied. After the loading is completed, if the large model does not perform reasoning, only a small amount of computing resources are required. Therefore, the current remaining resources of the server in this specific embodiment are the remaining resources after the large model is loaded on the server, and the remaining resources provide computing resources for the reasoning service of the large model. It should be emphasized that the computing resources of each server are far greater than the amount of resources occupied when the large model is reasoning.

[0193] For example, the initial time is Load a large model into On the servers, each server has four computing resources: graphics card G, video memory V, bandwidth B, and throughput T. The current remaining resources of the entire service cluster are expressed by the following formula:

[0194] ;

[0195] Represents the current remaining resources of the entire service cluster, where each row represents the remaining resources corresponding to the four computing resources of a server: graphics card G, video memory V, bandwidth B, and throughput T; Represents the first server in the server cluster; Represents the second server in the server cluster; Characterizes the first servers; Indicates the remaining resources of the graphics card of the first server; Indicates the remaining graphics card resources of the second server; Characterization The remaining graphics card resources of each server; Indicates the remaining video memory resources of the first server; Indicates the remaining video memory resources of the second server; Characterization The remaining video memory resources of each server; Characterizes the remaining bandwidth resources of the first server; Characterizes the remaining bandwidth resources of the second server; Characterization The remaining bandwidth resources of each server; Represents the throughput remaining resources of the first server; Represents the throughput remaining resources of the second server; Characterization The throughput remaining resources of the server.

[0196] After the large model is loaded on each server in the server cluster, each server can start receiving and processing inference requests. Since inference requests or types of inference requests are repeated, different computing resource allocation schemes need to be designed based on the inferred features.

[0197] In the first part above, the computing resource demand corresponding to each inference request and the request priority of the inference request are predicted. In the same time period, the inference requests are sorted according to the request priority corresponding to each inference request among multiple inference requests, and the inference requests are processed one by one according to the sorting results. However, when processing inference requests, one inference request needs to be processed on one server. Therefore, resource utilization and request processing efficiency need to be considered when selecting a server. Specifically, the selection of the server can be comprehensively considered in two sub-parts.

[0198] (1) Build server computing power indicators (server performance) to achieve rapid processing of inference requests, thereby accelerating resource release and improving the service efficiency of large models.

[0199] Considering that different servers are equipped with different graphics cards or processors, their computing rates and transmission speeds are often different, and each server's storage, network, networking methods, etc. will affect the server's computing power, that is, the server's server performance. Therefore, it is necessary to comprehensively consider the server's computing power indicators (specified performance indicators) and measure them from the perspective of the server's transaction processing. Generally, the performance of different online transaction processing systems can be evaluated according to traditional testing standards, thereby achieving a ranking of the server's server performance.

[0200] The traditional test standard is the TPC-C standard, which uses the following formula to calculate the transaction processing rate of a server, that is, the number of transactions processed by the server per minute.

[0201] ;

[0202] in, Characterizes the transaction processing rate of the server.

[0203] However, since the processing of inference requests by large models is different from traditional data processing, the improvement of the inference efficiency of each inference request by the processor (for example, GPU) on the server is much greater than that of traditional data processing. Therefore, the traditional test standard cannot accurately measure the server computing power index of the server. Therefore, the traditional TPC-C computing mode is optimized in the following ways:

[0204] (1-1) Determine different indicators that affect server performance (specify performance indicators). The above different indicators include: the transaction processing rate tpmC, GPU floating point performance (TF), GPU memory capacity (X) and GPU bandwidth (Y) of each server calculated using the TPC-C calculation model; wherein the above TF, X and Y can be directly determined based on the model of the GPU installed on the server.

[0205] (1-2) Data standardization. The range standardization method is used to eliminate the dimension effect, that is, the following formula is used:

[0206] ;

[0207] in, Characterizes the service cluster Server The standardized data corresponding to the indicators, Characterizes the service cluster Server The original data of the indicators; Represents the first The maximum raw data in the indicator; Represents the first The minimum raw data in an indicator.

[0208] (1-3) Use the hierarchical analysis method to determine the weights of different indicators. The goal of this step is to evaluate the server performance of each server in the service cluster, with the criteria of tpmC, TF, X and Y. Figure 3 The comparison judgment matrix shown in the figure compares each criterion pairwise. After obtaining the comparison judgment matrix, solve the maximum eigenvalue , and then obtain the After obtaining the corresponding eigenvector, the following formula is used to calculate the consistency index corresponding to the comparison judgment matrix;

[0209] ;

[0210] in, Characterize the consistency index corresponding to the comparison judgment matrix, Characterizes the matrix order of the comparison judgment matrix.

[0211] Then, the consistency ratio of the comparison judgment matrix is ​​calculated using the following formula;

[0212] ;

[0213] in, is the consistency index through literature or standard query, and the matrix order Related.

[0214] In this way, the consistency ratio is calculated After that, by determining The value is compared with the predetermined value 0.1 to determine whether the set comparison judgment matrix can meet the consistency requirement.

[0215] when When , it indicates that the comparison judgment matrix set does not meet the consistency requirements and needs to be redesigned; when When , it indicates that the set comparison judgment matrix meets the consistency requirement. At this time, The corresponding eigenvector is normalized, where the normalized eigenvector is a row vector, and each column of the row vector represents the weight corresponding to each criterion.

[0216] (1-4) Calculate the decision matrix. The decision matrix is ​​obtained by the weight matrix and the standardized matrix of each indicator, and the decision matrix is ​​calculated using the following formula;

[0217] ;

[0218] in, Characterization The weight of each indicator; Characterization Server The element value of the indicator in the decision matrix.

[0219] (1-5) Considering that the determined tpmC, TF, X and Y are all benefit indicators, that is, the larger the indicator, the stronger the server performance of the server. For each server, the closer the comprehensive value of all its indicators is to the maximum value of each column in the matrix (the maximum value is the maximum value of the weighted standard data of all indicators, that is, the maximum element value in the matrix), it indicates that the server performance of the server is good. Therefore, the maximum value of all columns in the decision matrix is ​​taken to form a positive ideal solution: ; and take the minimum value of all columns in the decision matrix to form a negative ideal solution: ; Then, the distance of each server's weighted normalized index to the positive ideal solution and the negative ideal solution, as well as the relative closeness of each server, are calculated using the following formula group; the above formula group is as follows:

[0220] ;

[0221] ;

[0222] ;

[0223] in, Characterization The distance from the weighted normalized index of each server to the positive ideal solution (first distance); Characterization The distance from the weighted normalized index of each server to the negative ideal solution (second distance); Characterization The relative closeness of each indicator of the server to the negative ideal solution. is a universal quantifier used to represent all or any, that is, the character It is used to indicate that a proposition is true for all members of a certain class of things. Indicates that the formula group is for servers from the first to the All servers are established.

[0224] From the above formula group, it can be seen that when the The smaller the distance between a server and the negative ideal solution, The closer it is to 0; when the distance from the positive ideal solution is smaller, The closer it is to 1; therefore, according to The size of The priority ranking of each server in the entire server cluster (server performance ranking).

[0225] As can be seen from step (1), the performance of the server is calculated by comprehensively considering the various hardware indicators of the server. Among them, the weights of different indicators obtained according to the actual application scenario can make the calculation results more in line with actual needs. So far, the server performance ranking and request priority ranking have been obtained, and then resources are allocated to each inference request.

[0226] Since inference requests need to be processed by a single server, it is necessary to comprehensively consider how to plan the four resources required for a request on the server cluster. Specifically, the four resources of graphics card G, video memory V, bandwidth B, and throughput T in the first part can be combined to design a specific allocation method.

[0227] (2-1) According to the first part, is the resource matrix, The current remaining resources of the server are ; Characterization The relative closeness of each indicator of each server to the negative ideal solution, that is, Characterization The overall performance of the servers. Characterization The four resources required for the inference request are If the inference request is assigned to Server processing, then , otherwise 0.

[0228] Based on the above situation, the constraints (selection conditions) can be determined as:

[0229] ;

[0230] in, Indicates the maximum occupancy ratio of computing resources of each server. The value limit can ensure that when allocating computing resources in each server, some computing resources are still reserved for handling emergencies to ensure that the server can maintain normal operation. Indicates that for the first inference request to the Inference requests, formula , all hold true; accordingly, the above characters For the traffic from the first server to the servers, the formula above , are all true.

[0231] At this point, based on the four resources required for each inference request, the constraints that need to be combined when allocating resources are determined.

[0232] (2-2) After determining the constraints, construct an objective function to determine the resource allocation method. Since the server comprehensive performance parameter (server performance) has been constructed previously, this parameter represents the performance level of a server. When processing inference requests, the improvement of inference efficiency means that the occupied resources are released faster. Therefore, when allocating resources, it is necessary not only to consider the balanced allocation of resources to meet more inference requests, but also to take into account the processing efficiency of inference requests to speed up resource release and ensure resource utilization from a higher perspective of resource release.

[0233] Therefore, the resource utilization effect under different allocation methods is measured by the comprehensive imbalance through the following objective function:

[0234] ;

[0235] in:

[0236] ;

[0237] In the above formula, It is used to represent the total resources required for inference requests allocated to each server on all servers, accounting for the The average of the current remaining resource ratios of the servers; The meaning of is the variance of the proportion of the demand for the four computing resources, and the maximum value is taken from the variance of the proportion of the four computing resources. This maximum value represents the upper limit of the deviation degree of the four computing resources; the right-hand side term The formula itself inherently means to find the mean of the target index, which is the server comprehensive parameters of all servers multiplied by the After the product of the current remaining resource occupancy ratios of the servers is calculated, the sum of the products of the occupancy ratios is added together; is the parameter that needs to be solved. Therefore, there will be multiple allocation situations of computing resources, and correspondingly there will be multiple allocation results. The formula that limits the right-hand side determines the largest of the multiple distribution results, that is, the final determination It is on the right side The formula within achieves the mean of the maximum and values.

[0238] From the overall perspective of the formula, if the maximum deviation of the four computing resources from the mean is smaller, it can be shown that the overall resource ratio is more balanced, that is, to make Minimum, (first weight) multiplied by the left term should be smaller, so that the deviation of the four computing resources will be smaller; in addition, Minimum, (Second weight) multiplied by the right side That is, only when more inference requests (or greater computing resource requirements) are processed by the server with the strongest server performance, the right-hand side item The larger the right side, the This indicates that servers with better performance are given priority in allocating more inference requests or requests that require more computing resources. However, when servers with better performance handle inference requests that require more computing resources, the resource deviation will increase. Therefore, by combining the left side, and The size of is used to focus on resource balancing or performance priority, and the allocation method of inference requests under the objective function is solved, that is, to determine which server in the service cluster each inference request is allocated to.

[0239] (2-3) The constructed objective function actually combines the two indicators of server performance and resource balance allocation, where the right-hand side term The formula in the table is used to characterize the performance of the server; the left side term The formula in is used to characterize the resource balancing index; when allocating the inference request using the solution provided in this specific embodiment, the integer programming problem method can be used to solve the objective function and the constraint conditions; specifically, as shown in step F:

[0240] Step F: Change the known items, , , , (Indicates the initial moment after the model is loaded. The remaining resources of the current servers are input into the objective function, and the objective function and constraints are solved using the integer programming method. This method may be iterated multiple times, where each iteration is actually a solution to a linear programming problem. Finally, the solution is obtained. The allocation results of inference requests , the distribution result is used to characterize the The inference request is assigned to servers.

[0241] Afterwards, according to the distribution results Update of resources used , and the inference requests entering in the next cycle (receiving cycle) are brought into the above objective function and constraints with the updated remaining resources of the service cluster and the computing resource requirements corresponding to each inference request in the next cycle, and the solution is continued according to the above step F.

[0242] However, it should be noted that when an inference request is processed, the resources occupied by the inference request will be released. At this time, the remaining resources of the entire service cluster will change. The matrix can detect whether there is resource release, and if so, update it using the following formula ;

[0243] ;

[0244] in, Characterization The remaining resources after the server update, Characterization The current remaining resources of the server, Characterization The server handles The amount of computing resources used for an inference request. Characterization The amount of resources released by the server.

[0245] In addition, considering that computing resources are limited, when there are many inference requests to be processed during the peak call period, all requests for a certain cycle cannot be processed. That is to say, the multiple unprocessed inference requests received in the embodiment of the present invention are inference requests received in at least one receiving cycle, and resource allocation cannot meet the constraint conditions. In the above case, the request priority of the inference request can be used to divide the requests in the entire cycle, and each inference request with a high priority level can be processed first, and the remaining requests are weighted and processed, and then the priority is rearranged with each inference request received in the next cycle, and the processing is cyclical, waiting for the peak call to end.

[0246] The solution provided in this specific embodiment constructs an objective function that combines performance processing and resource balancing, and solves the combined constraints using integer programming methods. This optimization method combines comprehensive performance parameters with resource surpluses, which can both utilize the server's computing resources and speed up the overall request processing efficiency. This effectively maximizes the resource value of computing resources for large model inference requests, improves computing resource utilization efficiency, ensures the stability of large model-related inference services, and further improves user experience.

[0247] Based on the above method embodiment, the present invention also provides a resource scheduling device for large model reasoning requests, such as Figure 5 FIG. 1 is a schematic diagram of a resource scheduling device for a large model inference request provided by an embodiment of the present invention. The device is applied to a scheduling server in a server cluster. The server cluster also includes multiple target servers, each of which is deployed with multiple large models. The device includes:

[0248] A request determination module 510, for determining each target reasoning request to be scheduled for resources;

[0249] The demand determination module 520 is used to determine the computing resource demand corresponding to each target reasoning request; wherein the computing resource demand corresponding to each target reasoning request is the resource amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource demand corresponding to each target reasoning request includes: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model for processing the target reasoning request when running in the candidate server;

[0250] The result determination module 530 is used to substitute the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request;

[0251] An allocation module 540 is used to allocate each target reasoning request to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model;

[0252] The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is:

[0253] ;

[0254] Characterize the total number of target servers in the service cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes the second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand.

[0255] As can be seen from the above, a resource scheduling method for large model inference requests provided by an embodiment of the present invention is applied to a scheduling server in a server cluster, wherein the server cluster further includes multiple target servers, and each target server is deployed with multiple large models. Before the scheduling server performs resource scheduling, first, a pre-trained resource prediction model is used to predict the amount of computing resources required when each inference request is responded to and processed, as the computing resource demand corresponding to the inference request, and the server performance ranking of each target server in the server cluster is pre-determined.

[0256] In this way, when performing resource scheduling, the scheduling server determines each target reasoning request to be scheduled, and the computing resource requirements corresponding to each target reasoning request. Therefore, the computing resource requirements corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the service cluster are substituted into the objective function to solve the allocation result of each target reasoning request, and based on the allocation result of each target reasoning request, the target server where the large model for responding to the target reasoning request is deployed is determined. In this way, for each target reasoning request, the target reasoning request is allocated to the target server represented by the corresponding allocation result, so that the target server that receives the target reasoning request determines the computing resource requirements corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource requirements, so as to process the target reasoning request through the deployed large model.

[0257] The computing resource requirements corresponding to each target reasoning request include: the graphics card parameters, video memory parameters, network bandwidth required by the candidate server for processing the target reasoning request, and the model throughput of the large model used to process the target reasoning request when it runs in the candidate server. In other words, when the scheduling server predicts the computing resource requirements, it takes into account the server's hardware graphics card resources and network bandwidth resources, that is, the graphics card parameters and video memory parameters of the server, as well as the network bandwidth of the server and the model throughput of the large model used to process the target reasoning request when it runs in the server.

[0258] Based on this, the solution provided by the embodiment of the present invention is applied. Based on the predicted computing resource demand corresponding to each target reasoning request and the server performance ranking of each target server in the server cluster, when computing resources are limited, by coordinating the graphics card resources and network bandwidth resources of the server cluster, after each target reasoning request is allocated, the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements in the actual reasoning scenario, so as to reduce the possibility of lack or waste of computing resources allocated to reasoning requests, thereby achieving the purpose of optimizing the usage of computing resources of servers in the server cluster.

[0259] Optionally, in one implementation, the server performance ranking of each target server in the server cluster is determined based on a plurality of specified performance indicators of the target server;

[0260] The method for determining the server performance ranking of each target server in the server cluster includes:

[0261] For each target server, performing predetermined data standardization processing on each designated performance indicator of the target server to obtain standardized data corresponding to each designated performance indicator of the target server, and determining weight parameters corresponding to each designated performance indicator of the target server based on the standardized data corresponding to each designated performance indicator;

[0262] Based on the weight parameters and standardized data corresponding to each designated performance indicator of each target server, a target matrix for server performance of the service cluster is generated; wherein each row of matrix elements in the target matrix is ​​used to characterize the server performance of a target server;

[0263] Determine the maximum value of each column in the target matrix, and combine to obtain a positive ideal solution for the target matrix;

[0264] Determine the minimum value of each column in the target matrix, and combine to obtain a negative ideal solution for the target matrix;

[0265] For each target server, determine the first distance between each designated performance indicator of the target server and the positive ideal solution, and the second distance between each designated performance indicator of the target server and the negative ideal solution, and determine the relative closeness corresponding to the target server based on the obtained first distance and second distance; wherein the relative closeness corresponding to the target server is the relative closeness between each designated performance indicator of the target server and the positive ideal solution, or the relative closeness to the negative ideal solution;

[0266] Based on the relative proximity corresponding to each target server, each target server is ranked to obtain a server performance ranking of each target server in the server cluster.

[0267] Optionally, in one implementation, the multiple specified performance indicators of each target server include: the transaction processing rate of the target server, the floating-point computing performance of the processor carried by the target server, the video memory capacity of the processor, and the bandwidth of the processor.

[0268] Optionally, in an implementation manner, the request determination module 510 is specifically configured to:

[0269] For each inference request among the received but unprocessed inference requests, determine the computing resource requirement corresponding to the inference request, and use the computing resource requirement corresponding to the inference request and the predetermined model weight of the large model used to process the inference request as input content, and input them into a pre-trained request priority determination model to obtain the priority of the inference request;

[0270] Based on the priority corresponding to each of the multiple inference requests, determine each target inference request to be resource scheduled from the multiple inference requests;

[0271] The method for determining the computing resource requirement corresponding to the inference request includes:

[0272] For each large model, the resource usage required by the large model when processing historical reasoning requests of the target type is obtained, and the resource usage of the large model when processing each historical reasoning request is obtained; wherein the historical reasoning request of the target type is a request of the same category as the reasoning request that has completed reasoning;

[0273] The resource usage of each large model when processing each historical inference request is input into the pre-trained resource prediction model to obtain the computing resource demand corresponding to the inference request.

[0274] Optionally, in one implementation, the received unprocessed multiple inference requests are inference requests received within at least one receiving cycle;

[0275] Before determining, from the multiple inference requests, each target inference request to be resource scheduled based on the priority corresponding to each inference request in the multiple inference requests, the method further includes:

[0276] Determine an inference request among the multiple inference requests that meets the target selection condition; wherein the target selection condition includes not belonging to an inference request received within a recent receiving cycle;

[0277] Increase the priority of the identified inference request.

[0278] Optionally, in an implementation manner, the device further includes:

[0279] A condition setting module, used to set a selection condition for a target server for processing each target reasoning request according to the computing resource demand; wherein the selection condition is used to limit each target reasoning request to correspond to a target server for processing the target reasoning request;

[0280] The result determination module 530 is specifically used for:

[0281] The computing resource demand corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the server cluster are substituted into the objective function, and combined with the selection conditions to determine the allocation result of each target reasoning request.

[0282] Optionally, in one implementation, graphics card parameters, video memory parameters, network bandwidth of each target server and model throughput of the large model when running on the server are different.

[0283] The embodiment of the present invention also provides a server, such as Figure 6 As shown, it includes a processor 601 , a communication interface 602 , a memory 603 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 communicate with each other via the communication bus 604 .

[0284] Memory 603, used for storing computer programs;

[0285] The processor 601 is used to implement any resource scheduling method for large model inference requests provided by the above-mentioned embodiments of the present invention when executing the program stored in the memory 603.

[0286] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0287] The communication interface is used for communication between the above server and other devices.

[0288] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0289] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0290] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned resource scheduling methods for large model inference requests are implemented.

[0291] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any resource scheduling method for large model inference requests in the above embodiments.

[0292] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0293] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0294] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, server embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0295] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A resource scheduling method for large model inference requests, characterized in that: The method is applied to a scheduling server in a server cluster, wherein the server cluster further includes a plurality of target servers, each of which is deployed with a plurality of large models. Determine each target reasoning request to be scheduled for resources; Determine the computing resource requirements corresponding to each target reasoning request; wherein the computing resource requirements corresponding to each target reasoning request are the amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource requirements corresponding to each target reasoning request include: graphics card parameters, video memory parameters, network bandwidth required by a candidate server for processing the target reasoning request, and model throughput of a large model for processing the target reasoning request when running in the candidate server; Substituting the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request; For each target reasoning request, the target reasoning request is allocated to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model; The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is: ； Characterize the total number of target servers in the server cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes a second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand; the first weight and the second weight are pre-set by the user.

2. The method according to claim 1, characterized in that The server performance ranking of each target server in the server cluster is determined based on a plurality of specified performance indicators of the target server; The method for determining the server performance ranking of each target server in the server cluster includes: For each target server, performing predetermined data standardization processing on each designated performance indicator of the target server to obtain standardized data corresponding to each designated performance indicator of the target server, and determining weight parameters corresponding to each designated performance indicator of the target server based on the standardized data corresponding to each designated performance indicator; Based on the weight parameters and standardized data corresponding to each specified performance indicator of each target server, a target matrix for server performance of the service cluster is generated; wherein each row of matrix elements in the target matrix is ​​used to characterize the server performance of a target server; Determine the maximum value of each column in the target matrix, and combine to obtain a positive ideal solution for the target matrix; Determine the minimum value of each column in the target matrix, and combine to obtain a negative ideal solution for the target matrix; For each target server, determine the first distance between each designated performance indicator of the target server and the positive ideal solution, and the second distance between each designated performance indicator of the target server and the negative ideal solution, and determine the relative closeness corresponding to the target server based on the obtained first distance and second distance; wherein the relative closeness corresponding to the target server is the relative closeness between each designated performance indicator of the target server and the positive ideal solution, or the relative closeness to the negative ideal solution; Based on the relative proximity corresponding to each target server, each target server is ranked to obtain a server performance ranking of each target server in the server cluster.

3. The method according to claim 2, characterized in that The multiple specified performance indicators of each target server include: the transaction processing rate of the target server, the floating point computing performance of the processor carried by the target server, the video memory capacity of the processor and the bandwidth of the processor.

4. The method according to claim 1, characterized in that: The determining of each target reasoning request to be scheduled for resources includes: For each inference request among the received but unprocessed inference requests, determine the computing resource requirement corresponding to the inference request, and use the computing resource requirement corresponding to the inference request and the predetermined model weight of the large model used to process the inference request as input content, and input them into a pre-trained request priority determination model to obtain the priority of the inference request; Based on the priority corresponding to each of the multiple inference requests, determine each target inference request to be resource scheduled from the multiple inference requests; The method for determining the computing resource requirement corresponding to the inference request includes: For each large model, the resource usage required by the large model when processing historical reasoning requests of the target type is obtained, and the resource usage of the large model when processing each historical reasoning request is obtained; wherein the historical reasoning request of the target type is a request of the same category as the reasoning request that has completed reasoning; The resource usage of each large model when processing each historical inference request is input into the pre-trained resource prediction model to obtain the computing resource demand corresponding to the inference request.

5. The method according to claim 4, characterized in that The received and unprocessed multiple inference requests are inference requests received within at least one receiving cycle; Before determining, from the multiple inference requests, each target inference request to be resource scheduled based on the priority corresponding to each inference request in the multiple inference requests, the method further includes: Determine an inference request among the multiple inference requests that meets the target selection condition; wherein the target selection condition includes not belonging to an inference request received within a recent receiving cycle; Increase the priority of the identified inference request.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: According to the computing resource demand, setting a selection condition for a target server for processing each target reasoning request; wherein the selection condition is used to limit each target reasoning request to correspond to a target server for processing the target reasoning request; Substituting the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request, includes: The computing resource demand corresponding to each target reasoning request, the current remaining resources of each target server, and the predetermined server performance ranking of each target server in the server cluster are substituted into the objective function, and combined with the selection conditions to determine the allocation result of each target reasoning request.

7. The method according to claim 1, characterized in that The graphics card parameters, video memory parameters, network bandwidth of each target server and the model throughput of the large model when running on the server are different.

8. A resource scheduling device for large model inference requests, characterized in that: A scheduling server applied to a server cluster, wherein the server cluster further comprises a plurality of target servers, each of which is deployed with a plurality of large models, wherein the device comprises: A request determination module, used to determine each target reasoning request to be scheduled for resources; A demand determination module is used to determine the computing resource demand corresponding to each target reasoning request; wherein the computing resource demand corresponding to each target reasoning request is the resource amount of computing resources required to be used when the target reasoning request is responded to and processed, which is predicted by using a pre-trained resource prediction model; the computing resource demand corresponding to each target reasoning request includes: graphics card parameters, video memory parameters, network bandwidth required by a candidate server for processing the target reasoning request, and model throughput of a large model for processing the target reasoning request when running in the candidate server; A result determination module is used to substitute the computing resource demand corresponding to each target reasoning request, the current remaining resource amount of each target server, and the predetermined server performance ranking of each target server in the server cluster into the objective function to solve and obtain the allocation result of each target reasoning request; An allocation module is used to allocate each target reasoning request to a target server represented by an allocation result of the target reasoning request, so that the target server receiving the target reasoning request determines the computing resource demand corresponding to the target reasoning request, and performs resource scheduling for the deployed large model according to the determined computing resource demand, so as to process the target reasoning request through the deployed large model; The allocation result of each target reasoning request represents the target server where the large model for responding to the target reasoning request is deployed; the objective function is: ； Characterize the total number of target servers in the server cluster; Represents the total number of inference requests for each target; Characterization server performance ranking of target servers in the server cluster; Characterization target reasoning request; To characterize the The row vector of the computing resource requirements corresponding to the target inference request, Each column represents a computing resource requirement; Characterization The current remaining resources of the target server; Represents the average of the target ratio values ​​corresponding to each target server. The target ratio value corresponding to each target server is the ratio of the total amount of computing resources required for the target inference request allocated to the target server to the current remaining resources of the target server; When the value of is 1, it indicates that The target inference request is assigned to the target server for processing. When the value of is 0, it indicates that The target inference request is not assigned to the target server for processing; Indicating that the resource utilization of each target server in the server cluster meets the predetermined resource utilization requirements, When the value is the smallest, the allocation result of each target reasoning request is obtained; Characterize the first weight; Characterizes a second weight; when the first weight is greater than the second weight, it characterizes that resource allocation balance is prioritized as the allocation target of the resource utilization demand; when the first weight is less than the second weight, it characterizes that server performance is prioritized as the allocation target of the resource utilization demand; the first weight and the second weight are pre-set by the user.

9. A server, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images