Large model resource allocation method and device, equipment and medium
Through the traffic gateway, the large-model resource request is proxyed, combined with resource requirements and historical usage, the amount of additional reserved resources is calculated, and the real-time dynamic adjustment of large-model resources is achieved, which solves the problems of poor resource allocation flexibility and low utilization in the existing technology. Improve system stability and resource utilization efficiency.
Patent Information
- Application Number
- CN202411096211.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
AI Technical Summary
The existing large-model resource allocation method relies on manual intervention, has poor flexibility, and is difficult to achieve real-time dynamic adjustment of resources, resulting in low resource utilization and insufficient system stability.
Through the traffic gateway, the large-model resource request interface of the target service is proxyed, the resource requirements and historical resource usage are obtained, the additional reserved resources are calculated, and the resource allocation is allocated based on these data to achieve real-time dynamic adjustment of resources.
It improves the flexibility of resource allocation in large-scale models, improves resource utilization, compresses the reserved cache size, enhances system stability, and reduces risks.
Smart Images

Figure CN119094472A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as large model resource allocation, and specifically to a large model resource allocation method, a large model resource allocation device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include natural language processing technology, computer vision technology, speech recognition technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention
[0004] The present disclosure provides a large model resource allocation method, a large model resource allocation device, an electronic device, a computer-readable storage medium, and a computer program product.
[0005] According to one aspect of the present disclosure, a large model resource allocation method is provided, including: proxying a large model resource request interface of a target business via a traffic gateway; obtaining a first resource amount and a second resource amount of the target business, the first resource amount representing the resource amount demand applied for by the target business, and the second resource amount indicating the historical resource amount usage of the target business; calculating a third resource amount of the target business based on the first resource amount and the second resource amount of the target business, the third resource amount representing the resource amount additionally reserved for the target business; and allocating large model resources to the target business using a traffic gateway based at least on the second resource amount and the third resource amount of the target business.
[0006] According to another aspect of the present disclosure, a large model resource allocation device is provided, including: an agent unit, configured to proxy the large model resource request interface of the target business via a traffic gateway; an acquisition unit, configured to acquire a first resource amount and a second resource amount of the target business, the first resource amount representing the resource amount demand applied for by the target business, and the second resource amount indicating the historical resource amount usage of the target business; a calculation unit, configured to calculate a third resource amount of the target business based on the first resource amount and the second resource amount of the target business, the third resource amount representing the resource amount additionally reserved for the target business; and a resource allocation unit, configured to allocate large model resources to the target business using the traffic gateway based on at least the second resource amount and the third resource amount of the target business.
[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.
[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.
[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.
[0010] According to one or more embodiments of the present disclosure, by proxying the large model resource request interface of the target business through the traffic gateway, and using the traffic gateway to allocate large model resources to the target business, the flexibility of large model resource allocation is improved, and the real-time dynamic adjustment of the resources allocated to the business is realized, and manual intervention is no longer required. By calculating the additional amount of resources (third amount of resources) reserved for the target business based on the resource demand (first amount of resources) applied by the target business and the historical resource usage of the target business (second amount of resources), and then determining the amount of resources allocated to the target business based on the second amount of resources and the third amount of resources, the utilization rate of large model resources is improved, the size of the reserved cache is compressed, the stability of the system is improved, and the risk is reduced.
[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0013] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;
[0014] Figure 2 A flowchart of a large model resource allocation method according to an embodiment of the present disclosure is shown;
[0015] Figure 3 A schematic diagram of a large model resource allocation system according to an embodiment of the present disclosure is shown;
[0016] Figure 4 A flowchart of a large model resource allocation method according to an embodiment of the present disclosure is shown;
[0017] Figure 5 A flowchart of a large model resource allocation method according to an embodiment of the present disclosure is shown;
[0018] Figure 6 A structural block diagram of a large model resource allocation device according to an embodiment of the present disclosure is shown; and
[0019] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.
[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.
[0023] In related technologies, a commonly used method is to limit the flow of large model services (for example, APIs). For example, by observing product traffic trends and relying on human experience, a portion of large model resources (for example, tokens) is fixedly allocated to other businesses in a certain period of time.
[0024] To solve the above problems, the present disclosure proxies the large model resource request interface of the target business through a traffic gateway, and uses the traffic gateway to allocate large model resources to the target business, thereby improving the flexibility of large model resource allocation and realizing real-time dynamic adjustment of resources allocated to the business, without the need for manual intervention. The additional amount of resources (third amount of resources) reserved for the target business is calculated based on the resource demand (first amount of resources) applied for by the target business and the historical resource usage of the target business (second amount of resources), and then the amount of resources allocated to the target business is determined based on the second amount of resources and the third amount of resources, thereby improving the utilization rate of large model resources, compressing the size of the reserved cache, improving the stability of the system, and reducing risks.
[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0026] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
[0027] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable execution of the methods of the present disclosure.
[0028] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.
[0029] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
[0030] The user can use client devices 101, 102, 103, 104, 105 and / or 106 to perform human-computer interaction. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.
[0031] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.
[0032] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.
[0033] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
[0034] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
[0035] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.
[0036] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.
[0037] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.
[0038] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.
[0039] Figure 1The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.
[0040] According to one aspect of the present disclosure, a large model resource allocation method is provided. Figure 2 The flowchart of the large model resource allocation method 200 according to the embodiment of the present disclosure is shown. The method 200 includes: step S201, proxying the large model resource request interface of the target business via the traffic gateway; step S202, obtaining the first resource amount and the second resource amount of the target business, the first resource amount represents the resource amount demand applied by the target business, and the second resource amount indicates the historical resource amount usage of the target business; step S203, based on the first resource amount and the second resource amount of the target business, calculating the third resource amount of the target business, the third resource amount represents the resource amount additionally reserved for the target business; and step S204, at least based on the second resource amount and the third resource amount of the target business, using the traffic gateway to allocate large model resources to the target business.
[0041] Therefore, by proxying the large model resource request interface of the target business through the traffic gateway and using the traffic gateway to allocate large model resources to the target business, the flexibility of large model resource allocation is improved, and the real-time dynamic adjustment of the resources allocated to the business is realized, and manual intervention is no longer required. By calculating the additional resource amount (third resource amount) reserved for the target business based on the resource amount demand (first resource amount) applied by the target business and the historical resource amount usage of the target business (second resource amount), and then determining the resource amount allocated to the target business based on the second resource amount and the third resource amount, the large model resource utilization rate is improved, the size of the reserved cache is compressed, the stability of the system is improved, and the risk is reduced.
[0042] Figure 3 A schematic diagram of a large model resource allocation system 300 according to an embodiment of the present disclosure is shown. In the system, multiple businesses 310 request a large model interface, which is proxied through a traffic gateway 320. The traffic gateway follows up with the resource allocation master control 330 to determine the flow limit of the resources allocated to it, and requests the large model 340 if it passes. The traffic gateway also records the traffic data 350 of each business in real time (for example, a token usage request), and is responsible for estimating and allocating resource volume requirements (Quota) in the next time window. The resource allocation master control reads the resource usage request of each business in real time, and performs resource allocation according to the large model resource allocation method disclosed in the present disclosure.
[0043] The first resource amount may be expressed as RequiredTPM, which indicates the resource amount (Token Per Minute, TPM) required for the target service application or guarantee. The first resource amount may be obtained by various methods, which are not limited here.
[0044] The second resource amount may be represented as UsedTPM, indicating the historical resource usage of the service. In some exemplary embodiments, UsedTPM may indicate the actual resource amount used after applying for RequiredTPM.
[0045] In some embodiments, step S202 may be performed at a first target time, and the first target time may be determined according to a preset time interval.
[0046] According to some embodiments, step S202, obtaining the first resource amount and the second resource amount of the target service may include: obtaining the actual used resource amount in multiple preset time windows before the first target moment; and sorting the actual used resource amounts in the multiple preset time windows to determine the preset percentile value to obtain the second resource amount corresponding to the first target moment. The third resource amount may represent the amount of resources additionally reserved for the target service at the first target moment.
[0047] Therefore, through the above method, a more reasonable second resource amount can be obtained as an estimate of the historical resource usage.
[0048] In an exemplary embodiment, the preset time window may be 1 minute, and the preset percentile value may be the 80th percentile value. That is, the actual resource usage within multiple 1 minutes before the first target time may be obtained, and then the actual resource usage may be sorted and the 80th percentile value may be taken as the second resource amount (e.g., UsedTPM).
[0049] According to some embodiments, the plurality of preset time windows may be divided by taking the first time interval as a unit, and any two preset time windows may not overlap. By dividing the plurality of preset time windows in the above manner, the final overall resource allocation effect can be further improved.
[0050] The third resource amount may be expressed as ReservedTPM, which indicates that, based on UsedTPM, redundant resources need to be reserved for the target service and cannot be occupied.
[0051] According to some embodiments, step S203, based on the first resource amount and the second resource amount of the target service, calculating the third resource amount of the target service may include: in response to determining that the second resource amount is less than the first resource amount, determining the third resource amount based on the second resource amount and a preset ratio. The sum of the second resource amount and the third resource amount is no more than the first resource amount.
[0052] Therefore, by determining that the second resource amount (e.g., UsedTPM) of the target business is less than the first resource amount (e.g., RequiredTPM), determining the third resource amount additionally reserved for the target business based on the preset ratio of the second resource amount, and then allocating large model resources to the target business based on the second resource amount (e.g., UsedTPM) and the third resource amount (e.g., ReservedTPM), while maintaining the current resource usage of the target business, an additional portion of resources is reserved for the target business to cope with fluctuations, and at the same time ensuring that the total resources allocated to the target business will not exceed the resource demand it has applied for, leaving spare resources for other businesses, thereby improving the overall resource utilization rate.
[0053] In an exemplary embodiment, the preset ratio may be, for example, 20%. It is understandable that the preset ratio may also be personalized in combination with the target service, or may be dynamically adjusted, which is not limited here.
[0054] In some embodiments, in step S203, in response to determining that the first resource amount (e.g., RequiredTPM) is less than the second resource amount (e.g., UsedTPM), that is, the resource amount actually used by the target business is greater than or equal to the resource amount required by the target business, the third resource amount (e.g., ReservedTPM) may be zero. In other words, if the target business has used more than the resource amount required, no additional resources will be reserved for it.
[0055] In some embodiments, in step S204, the sum of the second resource amount and the third resource amount can be determined as the large model resource amount allocated to the target business, or the large model resource amount allocated to the target business can be determined based on the second resource amount and the third resource amount in other ways.
[0056] According to some embodiments, the first target time may be determined based on the first time interval. Figure 4 As shown, the large model resource allocation method may also include: step S405, at a second target moment after the first target moment, based on the duration between the first target moment and the second target moment and the actual amount of resources used between the first target moment and the second target moment, calculating the estimated resource amount; step S406, based on the estimated resource amount and the second resource amount, determining the fourth resource amount corresponding to the second target moment; step S407, based on the first resource amount and the fourth resource amount of the target business, calculating the fifth resource amount of the target business, the fifth resource amount representing the amount of resources additionally reserved for the target business at the second target moment; and step S408, at least based on the second resource amount and the fifth resource amount of the target business, using the traffic gateway to allocate large model resources to the target business.
[0057] Understandably, Figure 4 The operations and effects of steps S401 to S404 in the above description can be referred to. Figure 2 The description of steps S201 to S204 in the process will not be repeated here.
[0058] Therefore, through the above method, the amount of resources that need to be additionally reserved for the target business is redetermined at the second target moment after (between) the first target moment determined by the first time interval, thereby achieving more refined resource allocation on a time scale and improving resource allocation efficiency.
[0059] In the above step S405, the current resource usage status can be estimated according to the duration between the first target moment and the second target moment and the amount of resources actually used during this period to obtain an estimated resource amount. Since the estimated resource amount takes into account the amount of resources actually used during this period that is closer to the current amount than the second resource amount (e.g., UsedTPM), it can provide a more real-time resource usage status.
[0060] According to some embodiments, the second target moment may be determined based on the second time interval, and the first time interval is an integer multiple of the second time interval. Step S405: at the second target moment after the first target moment, based on the duration between the first target moment and the second target moment and the actual amount of resources used between the first target moment and the second target moment, calculating the estimated amount of resources may include: dividing the actual amount of resources used by the duration, and then multiplying by a multiple of the first time interval and the second time interval to obtain the estimated amount of resources.
[0061] In an exemplary embodiment, the first time interval is 1 minute, and the second time interval is 10 seconds. At the 0th second at the beginning of each minute, the TPM 80th percentile value in the past multiple preset time windows (X minutes) is calculated as the benchmark for the current minute, that is, the second resource amount. Steps S405-S408 can be performed once at the 10th second, 20th second, 30th second, 40th second, and 50th second, respectively, that is, each of the 10th second, 20th second, 30th second, 40th second, and 50th second can be used as the second target duration. Taking the 20th second as an example, the number of marks used in the minute is * 6 / 2, that is, the number of marks used in the minute is used to estimate the number of marks used for the full minute. This value is compared with the above benchmark (that is, the second resource amount), and the larger one is taken as the fourth resource amount.
[0062] In the above step S406, based on the second resource amount (e.g., UsedTPM) and the estimated resource amount, a fourth resource amount that also represents the historical resource usage can be obtained. The fourth resource amount comprehensively utilizes the second resource amount and the estimated resource amount, while referring to the resource amount used in an earlier historical time period, and can also consider the current resource amount usage fluctuation, so it has a better effect.
[0063] According to some embodiments, determining a fourth resource amount corresponding to the second target time based on the estimated resource amount and the second resource amount may include: determining a larger value of the estimated resource amount and the second resource amount as the fourth resource amount.
[0064] Therefore, through the above method, on the one hand, sufficient large model resources can be allocated to the target business, and on the other hand, it can better adapt to the rapid changes in the amount of resources, especially to cope with the situation of increased resource usage.
[0065] In the above step S407, the fifth resource amount may be calculated based on the first resource amount and the fourth resource amount in a manner similar to step S203, wherein the fourth resource amount is similar to the second resource amount, and the fifth resource amount is similar to the third resource amount.
[0066] In the above step S408, the large model resource amount allocated to the target service may be determined based on the second resource amount and the fifth resource amount in a manner similar to step S204, wherein the fifth resource amount is similar to the third resource amount.
[0067] As described above, the method of the present disclosure can also be used to comprehensively manage and allocate resources for large model resource requests of multiple target services.
[0068] Return to step S201. In step S201, the large model resource request interface of multiple services may be proxied via a traffic gateway.
[0069] According to some embodiments, step S202, obtaining the first resource amount and the second resource amount of the target service may include: obtaining the first resource amount of each of the plurality of target services, wherein the sum of the first resource amounts of each of the plurality of target services is not greater than the total resource amount. Figure 5 As shown, the large model resource allocation method may also include: step S504, determining the amount of free resources based on the total amount of resources, the second amount of resources of each of the multiple target businesses, and the third amount of resources; and step S505, in response to determining that the second amount of resources of at least one target business among the multiple target businesses is greater than the first amount of resources, determining the sixth amount of resources of at least one target business based on the amount of free resources and the first amount of resources of the at least one target business, the sixth amount of resources being represented by the amount of resources occupied by the corresponding target task.
[0070] Understandably, Figure 5 The operations and effects of steps S501 to S503 and S506 in the above description can be referred to. Figure 2 The description of steps S201 to S204 in the process will not be repeated here.
[0071] Therefore, through the above method, when the system as a whole has extra spare resources, these resources can be allocated to the over-limit target business whose actual used resources are greater than the applied resource demand, thereby improving the resource utilization efficiency.
[0072] In the above step S504, the amount of free resources can be obtained by the following formula:
[0073] FreeTPM=TotalTPM-(∑UsedTPM+∑ReservedTPM)
[0074] Among them, FreeTPM may be the free resource amount, TotalTPM may be the total resource amount, ∑UsedTPM may be the sum of the second resource amounts of multiple target services, and ∑ReservedTPM may be the sum of the third resource amounts of multiple target services.
[0075] In the above step S505, for target services whose second resource amount (eg, UsedTPM) exceeds the first resource amount (eg, ReservedTPM), the sixth resource amount that needs to be preempted for the target service can be determined based on the free resource amount and the first resource amount of the target service.
[0076] According to some embodiments, step S505, in response to determining that the second resource amount of at least one target business among multiple target businesses is greater than the first resource amount, determining the sixth resource amount of at least one target business based on the spare resource amount and the first resource amount of at least one target business may include: for each target business among at least one target business, determining the sixth resource amount of the target business based on the spare resource amount, the ratio of the first resource amount of the target business to the sum of the first resource amounts of at least one target business, and a preset proportional coefficient.
[0077] Therefore, through the above method, a more reasonable value of resource amount can be allocated to each over-limit target service whose actual usage exceeds the applied resource amount, thereby improving the overall allocation efficiency.
[0078] In an exemplary embodiment, the sixth resource amount can be calculated by the following formula:
[0079] GrabbedTPM=FreeTPM*RequiredTPM / ∑Excess-limited-businessRequiredTPM*y%, wherein GrabbedTPM can represent the sixth resource amount, FreeTPM represents the free resource amount, RequiredTPM represents the first resource amount of the current target business, ∑Excess-limited-businessRequiredTPM represents the sum of the first resource amounts of all excess-limited businesses, and y% represents the preset proportional coefficient.
[0080] In some embodiments, the value of the preset proportional coefficient can be set according to demand. The preset proportional coefficient is set because some resources need to be reserved for near-line / offline services. And it is not necessary to supply all FreeRPM resources to preemption, which will also be wasteful, so a preemption upper limit is set.
[0081] In some embodiments, after deducting the sixth resource amount seized for each over-limit service from the free resource amount, a further remaining resource amount can be obtained. This resource amount can be used for the above-mentioned near-line / offline services. In an exemplary embodiment, the supply priority is that near-line is greater than offline. All accessed near-line / offline tasks need to set the expected resource amount. For near-line tasks, the amount of resources can be adjusted up and down within a range not exceeding the expected resource amount based on the historical resource usage of the near-line tasks (for example, the resource usage in the last first time interval). For offline tasks, the remaining resources after the near-line task allocation is completed can be evenly distributed to each offline task.
[0082] According to another aspect of the present disclosure, a large model resource allocation device is provided. Figure 6 The device 600 includes: an agent unit 610, which is configured to proxy the large model resource request interface of the target business via a traffic gateway; an acquisition unit 620, which is configured to acquire a first resource amount and a second resource amount of the target business, the first resource amount represents the resource amount demand applied for by the target business, and the second resource amount indicates the historical resource amount usage of the target business; a calculation unit 630, which is configured to calculate a third resource amount of the target business based on the first resource amount and the second resource amount of the target business, the third resource amount represents the additional resource amount reserved for the target business; and a resource allocation unit 640, which is configured to allocate large model resources to the target business using the traffic gateway based on at least the second resource amount and the third resource amount of the target business.
[0083] It is understood that the operations and effects of the units 610 to 640 in the device 600 can be referred to above. Figure 2 The description of steps S201 to S204 in the process will not be repeated here.
[0084] According to some embodiments, the acquisition unit may include: a first acquisition subunit, configured to acquire the actual amount of used resources in multiple preset time windows before the first target moment; and a first determination subunit, configured to sort the actual amount of used resources in the multiple preset time windows and determine a preset percentile value to obtain a second amount of resources corresponding to the first target moment. The third amount of resources may represent the amount of resources additionally reserved for the target business at the first target moment.
[0085] According to some embodiments, the plurality of preset time windows may be obtained by dividing the first time interval as a unit, and any two preset time windows may not overlap.
[0086] According to some embodiments, the first target moment may be determined based on the first time interval. The device may also include: a second calculation unit configured to calculate the estimated resource amount at the second target moment after the first target moment based on the duration between the first target moment and the second target moment and the actual amount of resources used between the first target moment and the second target moment; a first determination unit configured to determine the fourth resource amount corresponding to the second target moment based on the estimated resource amount and the second resource amount; a third calculation unit configured to calculate the fifth resource amount of the target business based on the first resource amount and the fourth resource amount of the target business, the fifth resource amount representing the amount of resources additionally reserved for the target business at the second target moment; and a second resource allocation unit configured to allocate large model resources to the target business using the traffic gateway based on at least the second resource amount and the fifth resource amount of the target business.
[0087] According to some embodiments, the second target time may be determined based on the second time interval, and the first time interval may be an integer multiple of the second time interval. The second calculation unit may include: a calculation subunit configured to divide the actual amount of used resources by the duration, and then multiply it by a multiple of the first time interval and the second time interval to obtain an estimated amount of resources.
[0088] According to some embodiments, the first determining unit may include: a second determining subunit configured to determine a larger value between the estimated resource amount and the second resource amount as the fourth resource amount.
[0089] According to some embodiments, the first calculation unit may include: a third determination subunit, configured to determine a third resource amount based on the second resource amount and a preset ratio in response to determining that the second resource amount is less than the first resource amount, wherein the sum of the second resource amount and the third resource amount is no more than the first resource amount.
[0090] According to some embodiments, the acquisition unit may include: a second acquisition subunit, configured to acquire the first resource amount of each of the multiple target services, the sum of the first resource amounts of each of the multiple target services is not more than the total resource amount. The above-mentioned device may also include (not shown in the figure): a second determination unit, configured to determine the free resource amount based on the total resource amount, the second resource amount of each of the multiple target services, and the third resource amount; and a third determination unit, configured to determine the sixth resource amount of each of the at least one target service based on the free resource amount and the first resource amount of each of the at least one target service in response to determining that the second resource amount of each of the multiple target services is more than the first resource amount, wherein the sixth resource amount is characterized as the resource amount occupied by the corresponding target task.
[0091] According to some embodiments, the third determination unit may include: a fourth determination subunit, configured to determine, for each target business of at least one target business, the sixth resource amount of the target business based on the amount of free resources, the ratio of the first resource amount of the target business to the sum of the first resource amounts of each of the at least one target business, and a preset proportional coefficient.
[0092] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0093] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.
[0094] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0095] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0096] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700. The input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a 702.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0097] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods, processes, and / or processes described above. For example, in some embodiments, these methods, processes, and / or processes may be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods, processes, and / or processes described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform these methods, processes, and / or processes in any other appropriate manner (e.g., by means of firmware).
[0098] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0102] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.
[0103] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0105] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.
Claims
1. A large model resource allocation method, comprising: Proxies the large model resource request interface of the target business through the traffic gateway; Acquire a first resource amount and a second resource amount of the target service, wherein the first resource amount represents a resource amount requirement applied for by the target service, and the second resource amount indicates a historical resource amount usage of the target service; Calculating a third resource amount for the target service based on the first resource amount and the second resource amount for the target service, where the third resource amount is represented by an additional resource amount reserved for the target service; as well as Based at least on the second resource amount and the third resource amount of the target service, the traffic gateway is used to allocate large model resources to the target service.
2. The method according to claim 1, wherein: Acquiring the first resource amount and the second resource amount of the target service includes: Obtaining actual resource usage within a plurality of preset time windows before a first target time; and The actual resource usage amounts in the plurality of preset time windows are sorted and then a preset percentile value is determined to obtain the second resource amount corresponding to the first target time. The third resource amount represents the amount of resources additionally reserved for the target service at the first target time.
3. The method according to claim 2, wherein: The multiple preset time windows are obtained by dividing the first time interval as a unit, and any two preset time windows do not overlap.
4. The method according to claim 2, wherein: The first target time is determined based on a first time interval, and the method further includes: At a second target time after the first target time, calculating an estimated resource amount based on a duration between the first target time and the second target time and an actually used resource amount between the first target time and the second target time; Determining a fourth resource amount corresponding to the second target time based on the estimated resource amount and the second resource amount; Calculating a fifth resource amount of the target service based on the first resource amount and the fourth resource amount of the target service, wherein the fifth resource amount represents an additional resource amount reserved for the target service at the second target time; and Based at least on the second resource amount and the fifth resource amount of the target service, the large model resources are allocated to the target service using the traffic gateway.
5. The method according to claim 4, wherein: The second target time is determined based on a second time interval, and the first time interval is an integer multiple of the second time interval. Wherein, at a second target moment after the first target moment, based on the duration between the first target moment and the second target moment and the actually used resource amount between the first target moment and the second target moment, calculating the estimated resource amount includes: The actually used resource amount is divided by the duration, and then multiplied by a multiple of the first time interval and the second time interval to obtain the estimated resource amount.
6. The method according to claim 4, wherein: Determining, based on the estimated resource amount and the second resource amount, a fourth resource amount corresponding to the second target time includes: A larger value between the estimated resource amount and the second resource amount is determined as the fourth resource amount.
7. The method according to any one of claims 1 to 6, wherein: Calculating a third resource amount of the target service based on the first resource amount and the second resource amount of the target service includes: In response to determining that the second resource amount is less than the first resource amount, the third resource amount is determined based on the second resource amount and a preset ratio, wherein a sum of the second resource amount and the third resource amount is no more than the first resource amount.
8. The method according to any one of claims 1 to 6, wherein: Acquiring the first resource amount and the second resource amount of the target service includes: Acquire a first resource amount of each of a plurality of target services, wherein the sum of the first resource amounts of each of the plurality of target services is no greater than the total resource amount, Wherein, the method further comprises: Determine the amount of available resources based on the total amount of resources, the second amount of resources and the third amount of resources of each of the plurality of target services; and In response to determining that the second resource amount of at least one target business among the multiple target businesses is greater than the first resource amount, based on the free resource amount and the first resource amount of the at least one target business, the sixth resource amount of the at least one target business is determined, and the sixth resource amount is represented as the resource amount occupied by the corresponding target task.
9. The method according to claim 8, wherein: In response to determining that the second resource amount of at least one target service among the multiple target services is greater than the first resource amount, determining a sixth resource amount of the at least one target service based on the spare resource amount and the first resource amount of the at least one target service includes: For each target business among the at least one target business, the sixth resource amount of the target business is determined based on the free resource amount, the ratio of the first resource amount of the target business to the sum of the first resource amounts of the at least one target business, and a preset proportional coefficient.
10. A large model resource allocation device, comprising: A proxy unit configured to proxy a large model resource request interface of a target service via a traffic gateway; An acquisition unit is configured to acquire a first resource amount and a second resource amount of the target service, wherein the first resource amount represents a resource amount requirement applied for by the target service, and the second resource amount indicates a historical resource amount usage of the target service; A first calculation unit is configured to calculate a third resource amount of the target service based on the first resource amount and the second resource amount of the target service, wherein the third resource amount is characterized as an additional resource amount reserved for the target service; as well as The first resource allocation unit is configured to allocate large model resources to the target business by using the traffic gateway based at least on the second resource amount and the third resource amount of the target business.
11. The device according to claim 10, wherein: The acquisition unit comprises: A first acquisition subunit is configured to acquire actually used resource amounts within a plurality of preset time windows before a first target time; and The first determining subunit is configured to sort the actually used resource amounts in the plurality of preset time windows and determine a preset percentile value to obtain the second resource amount corresponding to the first target time. The third resource amount represents the amount of resources additionally reserved for the target service at the first target time.
12. The device according to claim 11, wherein The multiple preset time windows are obtained by dividing the first time interval as a unit, and any two preset time windows do not overlap.
13. The device according to claim 11, wherein: The first target time is determined based on a first time interval, and the device further includes: A second calculation unit is configured to calculate an estimated resource amount at a second target moment after the first target moment based on a duration between the first target moment and the second target moment and an actually used resource amount between the first target moment and the second target moment; A first determining unit is configured to determine a fourth resource amount corresponding to the second target time based on the estimated resource amount and the second resource amount; A third calculation unit is configured to calculate a fifth resource amount of the target service based on the first resource amount and the fourth resource amount of the target service, wherein the fifth resource amount represents an additional resource amount reserved for the target service at the second target time; and The second resource allocation unit is configured to allocate large model resources to the target business using the traffic gateway based on at least the second resource amount and the fifth resource amount of the target business.
14. The device according to claim 13, wherein: The second target time is determined based on a second time interval, and the first time interval is an integer multiple of the second time interval. Wherein, the second calculation unit includes: The calculation subunit is configured to divide the actual amount of used resources by the duration, and then multiply the result by a multiple of the first time interval and the second time interval to obtain the estimated amount of resources.
15. The device according to claim 13, wherein: The first determining unit includes: The second determining subunit is configured to determine a larger value between the estimated resource amount and the second resource amount as the fourth resource amount.
16. The device according to any one of claims 10 to 15, wherein: The first computing unit comprises: The third determination subunit is configured to determine the third resource amount based on the second resource amount and a preset ratio in response to determining that the second resource amount is less than the first resource amount, wherein the sum of the second resource amount and the third resource amount is no more than the first resource amount.
17. The device according to any one of claims 10 to 15, wherein: The acquisition unit comprises: The second acquisition subunit is configured to acquire the first resource amount of each of the plurality of target services, wherein the sum of the first resource amounts of each of the plurality of target services is not greater than the total resource amount. Wherein, the device further comprises: A second determining unit is configured to determine the amount of free resources based on the total amount of resources, the second amount of resources of each of the plurality of target services, and the third amount of resources; and The third determination unit is configured to determine, in response to determining that the second resource amount of at least one target business among the multiple target businesses is greater than the first resource amount, a sixth resource amount of each of the at least one target business based on the free resource amount and the first resource amount of each of the at least one target business, wherein the sixth resource amount is characterized by the resource amount occupied by the corresponding target task.
18. The device according to claim 17, wherein: The third determining unit includes: The fourth determination subunit is configured to determine the sixth resource amount of the target business for each target business among the at least one target business based on the free resource amount, the ratio of the first resource amount of the target business to the sum of the first resource amounts of each of the at least one target business, and a preset proportional coefficient.
19. An electronic device, wherein: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; wherein The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.
21. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
