Resource allocation method and server cluster for large language model

Abstract

The invention provides a resource allocation method for a large language model and a server cluster, and the method can comprise the steps: obtaining a to-be-processed service; screening a to-be-selected server set with a specified proportion from a server cluster, and determining a resource vacancy rate of each to-be-selected server in the to-be-selected server set; determining a target server from the to-be-selected server set based on the resource quantity required by the to-be-processed service and the resource vacancy rate of the to-be-selected server; and processing the to-be-processed service by using the target server, the to-be-processed service being a service provided by a large model and / or other services assisting the large model. In the implementation mode, computing resources can be utilized more fully and reasonably.

Description

Technical Field

[0001] This application relates to the field of computing resource allocation technology, and more specifically, to a resource allocation method and server cluster for large language models. Background Technology

[0002] The application of large-scale language models (such as the GPT series and BERT series) in question-answering systems has become an important research direction. These models process large amounts of text data, learn the deep structure of language, and achieve the understanding and generation of human language, thereby providing accurate answers in question-answering systems. As the model size increases, they demonstrate greater ability to understand complex questions and generate high-quality answers, but they also bring challenges such as low resource utilization efficiency and high operating costs. Summary of the Invention

[0003] The purpose of this application is to provide a resource allocation method and server cluster for large language models, which can make fuller and more reasonable use of computing resources.

[0004] In a first aspect, the present invention provides a resource allocation method for a large language model, comprising: obtaining services to be processed; selecting a set of candidate servers of a specified proportion from a server cluster, and determining the resource vacancy rate of each candidate server in the set of candidate servers; determining a target server from the set of candidate servers based on the resource amount required by the services to be processed and the resource vacancy rate of the candidate servers; and using the target server to process the services to be processed, wherein the services to be processed are services provided by the large model and / or other services that assist the large model.

[0005] In the above implementation, when there is a service to be processed, a target server can be allocated based on the actual needs of that service. Furthermore, when selecting a target server, a subset of candidate servers can be selected from a large server cluster, thereby reducing the computational burden on assessing server resource availability. Finally, by combining the resource requirements of the service to be processed and the resource vacancy rate of each candidate server, the most suitable target server for processing the service is determined.

[0006] In an optional implementation, before using the target server to process the service to be processed, the method includes: determining the target service resource quantity for the service to be processed based on the resource quantity required by the service to be processed and the resource vacancy rate of the target server; wherein the target resource quantity is the resource provided by the target server for executing the service to be processed.

[0007] In the above implementation, the amount of target service resources suitable for the service to be processed can be determined by combining the amount of resources required by the service to be processed and the resource vacancy rate of the target server, thereby reducing resource waste.

[0008] In an optional implementation, the method further includes: based on the resource usage during the execution of the service to be processed by the target server; and dynamically adjusting the resources provided by the target server for executing the service to be processed based on the resource usage.

[0009] In the above implementation method, the amount of resources provided by the target server can be dynamically adjusted based on the actual usage of the service during its use, thereby reducing resource waste when resources are allocated reasonably.

[0010] In an optional implementation, the method further includes: receiving a first task, wherein the first task is a task of the service to be processed, or a task of another service running on the target server; adding the first task to a first queue, wherein the tasks in the first queue are executed sequentially; and adding the second task to a second queue after any second task fails to be processed, wherein the tasks in the second queue are executed according to a set time pattern, and the more times the second task fails, the longer the second task waits to be executed.

[0011] In the above implementation, different response efficiencies for different tasks can be achieved based on two queues.

[0012] In an optional implementation, before adding the first task to the first queue, the method further includes: identifying the service type to which the first task belongs; adding the first task to the first queue includes: adding the first task to the first queue if the service type to which the first task belongs is an interactive task; adding the first task to the third queue if the service type to which the first task belongs is a background task, wherein the priority of the third queue is lower than that of the first queue.

[0013] In the above implementation, different queues can be used to store different types of tasks, thereby providing different processing priorities based on different types, which can better adapt to actual processing efficiency requirements.

[0014] In an optional implementation, the method further includes: allocating a total resource pool for the first queue and the third queue, wherein the available resource range of the first queue is [-A, B], the available resource range of the third queue is [0, A], and the sum of the resources of A and B is the total resource pool.

[0015] In an optional implementation, the method further includes: detecting whether each service in the server cluster is running normally based on a preset liveness detection interface; and restarting the abnormally interrupted service if any service is detected to be abnormally interrupted.

[0016] In the above implementation, during the operation of the service running on the server, a preset liveness detection interface can be used to detect whether the service is running normally. If a liveness detection fails, the failed task can be actively restarted.

[0017] In an optional implementation, the method further includes: repeatedly calling the service to be processed based on a pre-written script and recording the resource usage of the service to be processed; and determining the required amount of resources for the service to be processed based on the resource usage.

[0018] In the above implementation method, the estimated resources required by the pending services can be realized by automatically calling the pending services, so that the determined amount of resources required by the pending services can be relatively more objective and accurate.

[0019] In an optional implementation, the method further includes: generating a resource usage table based on the usage of the server cluster; and determining abnormal content and the cause of the abnormal content based on the resource usage table.

[0020] In the above implementation, the resource usage of the service to be processed can be recorded, thereby automatically inferring abnormal content and the cause of the abnormal content.

[0021] In a second aspect, the present invention provides a server cluster, comprising: multiple servers; each server includes a processor and a memory, the memory storing machine-readable instructions executable by the processor, wherein when the server is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in any of the foregoing embodiments.

[0022] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the method described in any of the foregoing embodiments.

[0023] Fourthly, the present invention provides a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements the method described in any one of the foregoing embodiments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram illustrating the interaction between a server cluster and a terminal device provided in an embodiment of this application; Figure 2 A block diagram of a server provided in an embodiment of this application; Figure 3 A flowchart of a resource allocation method for a large language model provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] In the field of Natural Language Processing (NLP), the application of large-scale language models (such as the Generative Pre-trained Transformer (GPT) series and the Transformer-based Bidirectional Encoder Representations from Transformers (BERT) series) in question-answering systems has become an important research direction. These models process large amounts of text data to learn the deep structure of language, achieving understanding and generation of human language, and thus providing accurate answers in question-answering systems. As the model size increases, they demonstrate greater ability to understand complex questions and generate high-quality answers, but this also brings challenges such as low resource utilization efficiency and high operating costs.

[0029] However, there are some problems in the processing and calling of large models: (1) High accuracy and high resource consumption coexist. While improving the accuracy of question answering systems, existing large language models have very high requirements for computing resources (such as GPUs and TPUs) and storage resources. This leads to the need for a lot of hardware support in actual deployment, which increases costs. (2) In the entire platform system, not just one large model can do everything. Most platforms / products need other small models to assist, such as embeddings models, BERT models, etc. The deployment of these models will also compete with large models for resources when resources are limited. (3) The model hyperparameters are growing day by day: With the iteration and update of the model, the model hyperparameters have increased from 6B to 175B. There will be even larger hyperparameter models in the future. Even after quantization, it is not easy to deploy such a large model. It requires distributed deployment and large memory usage.

[0030] To address the problems encountered in the processing and invocation of large models, the following methods are typically used to alleviate these problems: (1) Model compression and optimization techniques are employed, such as model pruning, quantization, and distillation, to reduce the size of the model and computational requirements, thereby reducing resource consumption. (2) The number of users accessing the service is controlled by controlling the load of the service, but due to the different content of the questions and answers, the large model resources consumed are also uncertain, and the parameters are difficult to adjust dynamically.

[0031] Based on the above research, the embodiments of this application can provide a resource allocation method and server cluster for large language models, which can make fuller and more reasonable use of computing resources.

[0032] To facilitate understanding of this embodiment, the operating environment for implementing the resource allocation method for a large language model disclosed in this application embodiment will first be described in detail.

[0033] like Figure 1 The diagram shown is a schematic diagram of the interaction between the server cluster 100 provided in the embodiment of this application.

[0034] Server cluster 100 may have multiple servers 110. These multiple servers 110 may include server nodes as processing devices, or a central server 110 that coordinates and allocates resources to various services. The central server 110 communicates with one or more server nodes via a network for data communication or interaction.

[0035] The central server 110 can be used to allocate resources for various services that require processing by the server cluster 100. For example, these services can be those provided by the large language model or those provided by models that assist the large language model.

[0036] Server 110 in server cluster 100 can be a network server, database server, etc.

[0037] During operation, the server cluster 100 can execute various steps in the resource allocation method for large language models provided in the embodiments of this application.

[0038] In one example, a use case for resource allocation methods for large language models could be intelligent question answering. In this scenario, the front-end terminal device 200 provides a consultation interface. After obtaining a question through this interface, the server cluster 100 can perform relevant calculations on the question. During the calculation process, the model involved in the intelligent question answering scenario can be used to determine the answer to the question. The model involved in the intelligent question answering scenario can run on various server nodes within the server cluster 100.

[0039] like Figure 2 The diagram shown is a block illustration of an electronic device. Server 110 may include memory 111 and processor 113. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of server 110. For example, server 110 may also include components that are more complex than those shown in the diagram. Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0040] The memory 111 and processor 113 described above are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The processor 113 described above is used to execute executable modules stored in the memory.

[0041] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the server 110 as defined in any embodiment of this application can be applied to the processor 113, or implemented by the processor 113.

[0042] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0043] The peripheral interface 114 described above couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 can be implemented on a single chip. In other instances, they can be implemented on separate chips.

[0044] The input / output unit 115 described above is used to provide user input data. The input / output unit 115 may be, but is not limited to, a mouse and keyboard, etc.

[0045] The aforementioned display unit 116 provides an interactive interface (e.g., a user interface) between the server 110 and the user, or displays image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display (LCD) or a touch display. If it is a touch display, it can be a capacitive touchscreen or a resistive touchscreen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more locations on the touch display and pass the sensed touch operations to the processor for calculation and processing.

[0046] The server 110 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the resource allocation method for large language models is described in detail below through several embodiments.

[0047] Please see Figure 3 This is a flowchart of a resource allocation method for a large language model provided in an embodiment of this application. The resource allocation method for a large language model provided in this embodiment can be applied to a server cluster, through which the steps in the resource allocation method for a large language model are executed. The following will describe... Figure 3 The specific process shown will be explained in detail.

[0048] Step 310: Obtain the service to be processed.

[0049] Taking the intelligent question answering scenario as an example, the service to be processed can be a service provided by one of the multiple deep learning models provided to achieve intelligent question answering.

[0050] Step 320: Select a specified proportion of candidate servers from the server cluster and determine the resource vacancy rate of each candidate server in the candidate server set.

[0051] The specified ratio here can be determined based on the number of servers in the server cluster. For example, if the number of servers in the server cluster is relatively large, the specified ratio can be set to a relatively small value. For instance, if the number of servers in the server cluster is one hundred, the specified ratio could be one-tenth, one-fifteenth, etc. Conversely, if the number of servers in the server cluster is relatively small, the specified ratio can be set to a relatively large value. For instance, if the number of servers in the server cluster is thirty, the specified ratio could be one-third, one-quarter, etc. Of course, depending on the actual needs, the specified ratio can be set to other different values ​​as needed. It is understood that setting this specified ratio can reduce the overall computational load while achieving a balanced distribution of computing resources.

[0052] Optionally, the set of candidate servers can be values ​​randomly selected from the server cluster. Random selection can improve the balance of the probability of each server in the server cluster being selected, and reduce the situation where some servers are overused and some servers are idle for a long time.

[0053] Step 330: Based on the amount of resources required for the service to be processed and the resource vacancy rate of the candidate servers, determine the target server from the candidate server set.

[0054] Resource vacancy rate can include the vacancy rate of resources such as memory, GPU, CPU, and video memory.

[0055] Optionally, the server with the highest resource vacancy rate can be selected as the target server.

[0056] Optionally, the required resources may include the demand for various resources such as memory, GPU, and CPU. The candidate server group identifies a portion of servers whose resources meet the required amount, and then selects servers with more spare capacity from this portion as the target servers.

[0057] For example, some large models typically have high VRAM usage but low GPU utilization. Therefore, servers running large models may have some GPUs with high idle rates. In this case, by detecting parameters such as VRAM and utilization on the current server, other tasks can be dynamically allocated to that server. This allows for the configuration of other services on servers running large models, ensuring full utilization of server resources.

[0058] Step 340: Use the target server to process the pending service.

[0059] Among them, the services to be processed are the services provided by the large model and / or other services that assist the large model.

[0060] Through steps 310 to 340 above, when a service needs to be processed, a target server can be allocated based on the actual needs of that service. Furthermore, when selecting a target server, a subset of candidate servers can be selected from a large server cluster, thereby reducing the computational burden on assessing server resource availability. Finally, by combining the resource requirements of the service to be processed and the resource vacancy rate of each candidate server, the most suitable target server for processing the service is determined.

[0061] Before step 340, the resource allocation method for large language models in this embodiment may further include: determining the target service resource quantity for the service to be processed based on the resource quantity required by the service to be processed and the resource vacancy rate of the target server.

[0062] The target resource quantity refers to the resources provided by the target server for executing the service to be processed.

[0063] The target service resource quantity can include the configuration amount of resources such as memory and processor.

[0064] Optionally, if the resources on the target server are relatively sufficient, resources can be allocated to the service to be processed according to the requested resource amount determined by the resource amount required by the service to be processed.

[0065] Optionally, when resources on the target server are relatively scarce, a value can be calculated based on the requested resource amount determined by the resource amount required by the service to be processed and the resource vacancy rate of the target server, and this value can be used to allocate resources to the service to be processed.

[0066] For example, a certain percentage of the target server's resource vacancy rate can be calculated first, and then compared with the requested resource amount determined by the resource amount required by the service to be processed. The smaller value is taken as the target service resource amount for the service to be processed. The certain percentage of the target server's resource vacancy rate can be a pre-set percentage, for example, a certain percentage can be identified as 50%, 60%, 40%, etc.

[0067] To make fuller use of the target server's resources and avoid wasting resources, the amount of resources available for the service to be processed can be dynamically adjusted by observing its performance on the target server after it has been running for a period of time.

[0068] The resource allocation method for large language models in this embodiment may further include steps 351 and 352.

[0069] Step 351: Based on the resource usage during the execution of the pending service on the target server.

[0070] Optionally, the resource usage of the target server during the execution of the pending service can be obtained at set intervals to determine whether the resource configuration of the pending service needs to be adjusted.

[0071] The set interval can be based on the operating frequency of the service to be processed. For example, if the operating frequency of the service to be processed is high, the set interval can be relatively short, such as 12 hours or 1 day. For example, if the operating frequency of the service to be processed is low, the set interval can be relatively long, such as 3 days or 5 days.

[0072] Step 352: Based on resource usage, dynamically adjust the resources provided in the target server for executing the pending services.

[0073] Optionally, a specified multiple of the resource usage can be calculated, and then the resource amount obtained after the specified multiple is compared with the target service resource amount. The smaller value can be selected as the updated implementation configuration resource amount for the service to be processed. The specified multiple can be a positive integer greater than 1, for example, a value of 2, 3, etc.

[0074] In one instance, after the service to be processed runs on the target server for a period of time, the resource usage during this period can be recorded. It was found that the average memory usage of the service to be processed was only 1GB. However, the memory allocation in the target service's resource pool was 30GB, indicating significant memory waste. Based on this, with a specified multiplier of 2, the memory usage of the service to be processed can be adjusted to 2GB.

[0075] In one instance, after the service to be processed runs on the target server for a period of time, the resource usage during this period can be recorded, revealing that the average memory usage of the service to be processed is only 8GB. However, the memory allocation in the target service's resource pool is 15GB. Based on this, by specifying a multiplier of 2, the memory usage of the service to be processed can be maintained at 15GB.

[0076] Based on the above logic, during the service usage process, the amount of resources provided by the target server can be dynamically adjusted based on its actual usage, thereby reducing resource waste when resources are allocated reasonably.

[0077] In this embodiment, to better respond to various tasks, different tasks can be responded to based on actual needs. The resource allocation method for large language models in this embodiment may further include: Step 361, Receive the first task.

[0078] The first task is either a service to be processed, or a task of another service running on the target server.

[0079] Step 363: Add the first task to the first queue.

[0080] In this embodiment, tasks in the first queue are executed sequentially. If available computing resources exist on the target server, tasks in the first queue can be executed first.

[0081] Step 365: After any second task fails to be processed, add it to the second queue.

[0082] In high-concurrency scenarios, when multiple users make simultaneous requests, this provides a guarantee to ensure that the large model service can provide data normally.

[0083] In the second queue, tasks are executed according to a set time pattern. The more times a second task fails, the longer it waits to be executed.

[0084] The second queue can serve as a dead-letter queue, which means that in the event of an exception during a large model request, the task for handling the exception is placed in the dead-letter queue.

[0085] Tasks in the second queue can be executed a specified number of times. If a task fails to execute successfully after the specified number of attempts, it can be recorded in the failed task database. If a task executes successfully within the specified number of attempts, the execution result can be sent back to the terminal device.

[0086] For example, the specified number of times can be set to a small value as needed, such as 3, 4, 5, etc.

[0087] Tasks in the second queue employ a backoff execution mechanism. To avoid frequent failed task initiations, tasks can be retried after a set interval following a failure. Optionally, Fibonacci numbers can be used to trigger multiple attempts to execute tasks in the second queue. For example, after the first failure, a 3-second retry can be initiated; if it still fails, a 5-second retry can be initiated; and if it fails again, an 8-second retry can be initiated.

[0088] To improve user experience and enable users to receive responses more promptly, after the target server receives the task request from the terminal device, it can first send a confirmation message to the terminal device. The target server can then place the task in the first queue mentioned above for computation.

[0089] To better adapt to the processing needs of different types of tasks, different response efficiencies can be provided for different types of tasks. Based on this, before step 363, the method of this embodiment may also include step 362, identifying the service type to which the first task belongs.

[0090] Optionally, classification can be based on actual usage scenarios and response requirements for various tasks.

[0091] Taking the resource allocation method for large language models as an example in the intelligent question answering scenario, tasks that require feedback to the user and are displayed to the user can be classified as one category, and background tasks that do not need to be displayed to the user can be classified as another category. Thus, tasks that require feedback to the user and are displayed to the user can be classified as interactive tasks, and background tasks that do not need to be displayed to the user can be classified as background tasks.

[0092] Step 363 above includes: adding the first task to the first queue if the service type of the first task is interactive task.

[0093] Step 364: If the service type of the first task is a background task, add the first task to the third queue.

[0094] In this process, the third queue has a lower priority than the first queue. Given the same computing resources, tasks in the first queue are executed first when the third queue has a lower priority than the first queue. For example, if tasks exist in the first queue, those tasks are executed first; if no tasks exist in the first queue, tasks in the second queue can be executed.

[0095] Optionally, a total resource pool can be pre-allocated for the first queue and the third queue, wherein the available resource range of the first queue is [-A, B], the available resource range of the third queue is [0, A], and the sum of the resources of A and B is the total resource pool.

[0096] In one instance, the total resource pool quota allocated to the first and third queues can be 32k (tokens), and the values ​​of A and B can both be 16k. This also results in the available resource range of the first queue being [-16k, 16k], which means that the target server's -16k can be overdrawn. The available resource range of the third queue is [0, 16k]. This means that if there are many tasks in the first queue, the quota resource will drop below 0, at which point the tasks in the second queue can no longer be executed.

[0097] In this embodiment, if a task fails to be processed in the first queue or the third queue, it can be added to the second queue to delay the repeated execution of the failed task.

[0098] To improve the effectiveness of tasks running on the server, running services can be monitored to promptly detect services that have been abnormally interrupted. Based on this, the method in this application embodiment may include: detecting whether each service in the server cluster is running normally based on a preset liveness detection interface; and restarting the abnormally interrupted service if any service is detected to be abnormally interrupted.

[0099] Optionally, if multiple attempts to probe a service fail, the service can be determined to be in an interrupted state.

[0100] For example, when registering a service to be processed, a preset liveness detection interface can be reserved. The preset liveness detection interface can periodically detect whether the current service to be processed is still alive.

[0101] If multiple liveness detection attempts fail, the system will determine that the current service is unavailable and will automatically restart the model service via command.

[0102] To enable relevant personnel to better understand the operation of the service, the resource allocation method for large language models in this application embodiment may further include: Step 371: Generate a resource usage table based on the usage of the server cluster.

[0103] For example, a resource usage table can include the usage of various resources at different times.

[0104] For example, the resource usage table can be organized by service, recording the usage of various resources by each service at different times.

[0105] For example, the resource usage table can be based on servers, recording the usage of various resources by each server in the server cluster at different times.

[0106] Step 372: Determine the abnormal content and the cause of the abnormal content based on the resource usage table.

[0107] In one instance, the current resource usage table shows that a server's GPU utilization is very low. Based on this, log analysis reveals that recent road construction caused a cable to be severed, rendering tasks unreachable and resulting in no requests being sent to that server. A possible recommendation for this anomaly is: Urge the supplier to perform repairs as soon as possible; contact information is: xxx.

[0108] In this embodiment, to make the resource allocation of each service more reasonable, the amount of resources required by the service to be processed can be estimated before allocation. Based on this, the following steps may be included before step 310.

[0109] Step 410: Based on the pre-written script, call the service to be processed in a loop and record the resource usage of the service to be processed.

[0110] For example, a pre-written script can repeatedly call the inference interface of the service to be processed. The bag-of-record processing service can record the usage of various resources during the invocation process.

[0111] Step 420: Based on resource usage, determine the amount of resources required for the service to be processed.

[0112] For example, by continuously calling the method in step 410 for a period of time, the average and peak usage of various resources can be determined. This allows for the determination of the amount of resources required to meet peak usage.

[0113] Optionally, the GPU type and quantity decision rule can be based on the peak value observed in step 410 multiplied by a first coefficient to determine the minimum video memory requirement per instance. The first coefficient can be a value greater than 1 but less than 3, for example, it could be 1.2, 1.3, 1.5, 2, 2.5, etc. This represents the minimum video memory requirement per instance.

[0114] Optionally, the required memory can be determined by multiplying the sum of the memory required when loading the model corresponding to the service to be processed and the peak memory usage during execution by a second coefficient. For example, the second coefficient can be a value greater than 1 but less than 3, such as 1.2, 1.3, 1.5, 2, 2.5, etc. The number of CPU cores can be multiplied by a third coefficient based on the number of GPU cards. The third coefficient can be an integer, such as 4, 5, 6, 7, 8, etc.

[0115] To improve resource efficiency during execution, adaptive elastic scaling threshold rules can be pre-configured. For example, resource allocation can be dynamically adjusted by multiplying the service load parameters provided by the large model by a fourth coefficient. This fourth coefficient can be an integer, such as 2, 3, or 4.

[0116] Optionally, the low utilization rate of each resource in the server cluster can be marked at each set period, and the resources of the marked low-utilization servers and their corresponding services can be reduced at a specified time node. For example, the low utilization rate can be a value less than 50%, such as 20%, 30%, or 40%.

[0117] The specified time point can be a time point that the service to be processed is not frequently used, such as 2:00 AM or 3:00 AM every day.

[0118] Optionally, step 410 can be performed on a test device. To better ensure the safety of the test device, the test in step 410 can be dynamically terminated when the utilization rate of the test device and the usage of video memory reach 85%.

[0119] Based on the resource allocation method for large language models described above, the provided resource configuration is more granular, allowing for more efficient utilization of large model resources. Resource control is highly effective in high-concurrency and multi-scenario applications, requiring only configuration input or simple priority definition. Furthermore, resource allocation can be dynamically adjusted based on large model instances and server load, eliminating the need for manual settings. Additionally, it possesses capabilities such as service registration control, health checks (liveness detection), circuit breaking, and rate limiting, making it more adaptable to various scenarios, such as offline computing, real-time computing, and traffic control.

[0120] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the resource allocation method for a large language model described in the above method embodiments.

[0121] The computer program product for the resource allocation method for a large language model provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the resource allocation method for a large language model described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0123] In addition, the method steps in the various embodiments of this application can be integrated together to form an independent part for execution, or each method step can be executed by a separate module, or two or more steps can be formed into an independent part for execution.

[0124] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0125] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for resource allocation for large language models, characterized in that, The method comprises: obtaining a to-be-processed service; selecting a specified proportion of candidate server sets from a server cluster, and determining a resource vacancy rate of each candidate server in the candidate server set; determining a target server from the candidate server set based on the required resource amount of the to-be-processed service and the resource vacancy rate of the candidate server; processing the to-be-processed service using the target server, wherein the to-be-processed service is a service provided by a large model and / or other services assisting the large model.

2. The method of claim 1, wherein, Before the processing of the to-be-processed service using the target server, the method comprises: determining a target service resource amount for the to-be-processed service based on the required resource amount of the to-be-processed service and the resource vacancy rate of the target server; wherein the target resource amount is the resource provided by the target server for executing the to-be-processed service.

3. The method of claim 1, wherein, The method further comprises: determining the resource usage amount of the target server during the execution of the to-be-processed service; dynamically adjusting the resource provided by the target server for executing the to-be-processed service based on the resource usage amount.

4. The method of claim 1, wherein, The method further comprises: receiving a first task, wherein the first task is a task of the to-be-processed service or a task of other services running on the target server; adding the first task to a first queue, wherein the tasks in the first queue are executed in order; after any second task fails to be processed, adding the second task to a second queue, wherein the tasks in the second queue are executed according to a set time rule, and the more times the second task fails, the longer the second task waits to be executed.

5. The method of claim 4, wherein, Before the adding of the first task to the first queue, the method further comprises: identifying the service type to which the first task belongs; the adding of the first task to the first queue comprises: when the service type to which the first task belongs is an interactive task, adding the first task to the first queue; when the service type to which the first task belongs is a background task, adding the first task to a third queue, wherein the priority of the third queue is lower than that of the first queue.

6. The method of claim 5, wherein, The method further comprises: allocating a total resource pool for the first queue and the third queue, wherein the available resource range of the first queue is [-A, B], the available resource range of the third queue is [0, A], and the sum of the resources of A and B is the total resource pool.

7. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: detecting whether each service in the server cluster is in normal operation based on a preset live detection interface; in the case of detecting that any service is abnormally interrupted, restarting the abnormally interrupted service.

8. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: cyclically calling the to-be-processed service based on a pre-written script and recording the resource usage of the to-be-processed service; determining the required resource amount of the to-be-processed service based on the resource usage.

9. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: generating a resource usage table for the usage of the server cluster; determining abnormal content and the cause of the abnormal content based on the resource usage table.

10. A server cluster, characterized by The method comprises: a plurality of servers; The server comprises a processor, a memory, the memory stores machine readable instructions executable by the processor, when the server runs, the machine readable instructions are executed by the processor to perform the steps of the method in any one of claims 1 to 9.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is run by the processor, the steps of the method in any one of claims 1 to 9 are executed.

12. A computer program product, characterised in that, The computer program product comprises a computer program, when the computer program is executed by the processor, the method in any one of claims 1 to 9 is implemented.

Images