Model resource deployment method and device, equipment, storage medium and program product

Abstract

The invention discloses a model resource deployment method and device, equipment, a storage medium and a program product, relates to the technical field of communication, and aims to effectively utilize GPU resources. The method comprises the following steps: acquiring a reasoning service demand; estimating GPU resources required by a large model to be deployed according to the reasoning service requirements; required GPU resources are deployed; and under the condition that the reasoning task is received, adjusting the deployed GPU resources according to the reasoning task. According to the embodiment of the invention, GPU resources can be effectively utilized.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a model resource deployment method, device, equipment, storage medium and program product. Background Art

[0002] Large model deployment refers to the process of applying a trained large model to an actual production environment. Graphics Processing Unit (GPU) resources are designed for large-scale parallel processing of complex and data-intensive computing tasks and are widely used in many fields such as games, scientific computing, artificial intelligence, virtual reality, and deep learning.

[0003] The prior art provides a method for scheduling and deploying deep learning models under limited hardware resource conditions. In this method, the reasoning task is divided into stages, and resource evaluation is performed for each stage, and resource scheduling is performed for each stage of the reasoning task. Since this method requires resource scheduling for each stage of the reasoning task, GPU resources cannot be effectively utilized. Summary of the invention

[0004] Embodiments of the present application provide a model resource deployment method, apparatus, device, storage medium, and program product to effectively utilize GPU resources.

[0005] In a first aspect, an embodiment of the present application provides a model resource deployment method, including:

[0006] Obtain inference business requirements;

[0007] Estimate the GPU resources required for the large model to be deployed based on the inference business requirements;

[0008] Deploy the required GPU resources;

[0009] When an inference task is received, the deployed GPU resources are adjusted according to the inference task.

[0010] Optionally, the inference business requirements include business requirements, model requirements of the large model, and GPU information;

[0011] The estimating the graphics processing unit (GPU) resources required for the large model to be deployed according to the inference business requirements includes:

[0012] Determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model, and the GPU information;

[0013] According to the business requirements and the GPU resources required by the model instance, estimate the GPU cluster resources corresponding to the large model.

[0014] Optionally, the model requirement includes one or more of the model scale, model structure, input token length, and output token length; the business requirement includes business latency requirement; the GPU information includes the size of the GPU graphics card;

[0015] The determining, according to the business requirements, the model requirements of the large model and the GPU information, the GPU resources required for the single model instance includes:

[0016] Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token;

[0017] The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

[0018] Optionally, the method further includes:

[0019] According to the service delay requirement, a maximum batch size (batchsize) that meets the service delay requirement is determined.

[0020] Optionally, estimating GPU cluster resources corresponding to the large model according to the business requirements and the GPU resources required by the model instance includes:

[0021] According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained;

[0022] Determine a networking method according to the total number;

[0023] The parallel mode of the large model is determined according to the total number and the form of the GPU card.

[0024] Optionally, the method further includes:

[0025] Send query requests to the intelligent computing resource pool through the query interface;

[0026] Receive the GPU information sent by the intelligent computing resource pool.

[0027] Optionally, the method further includes:

[0028] Sending a first message to the service requester, wherein the first message includes information about estimating GPU resources;

[0029] A first response sent by the service requester is received, and the estimated GPU resources are adjusted according to the first response.

[0030] Optionally, when receiving an inference task, adjusting the deployed GPU resources according to the inference task includes:

[0031] If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

[0032] In a second aspect, an embodiment of the present application provides a model resource deployment device, including:

[0033] A first acquisition module is used to acquire reasoning business requirements;

[0034] A first determination module is used to estimate the GPU resources required for the large model to be deployed according to the inference business requirements;

[0035] A first processing module, used to deploy required GPU resources;

[0036] The second processing module is used to adjust the deployed GPU resources according to the inference task when receiving the inference task.

[0037] Optionally, the inference business requirements include business requirements, model requirements of the large model, and GPU information;

[0038] The first determining module includes:

[0039] A first determination submodule is used to determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model and the GPU information;

[0040] The second determination submodule is used to estimate the GPU cluster resources corresponding to the large model according to the business requirements and the GPU resources required by the model instance.

[0041] Optionally, the model requirement includes one or more of the model scale, model structure, input token length, and output token length; the business requirement includes business latency requirement; the GPU information includes the size of the GPU graphics card;

[0042] The first determining submodule is used to:

[0043] Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token;

[0044] The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

[0045] Optionally, the first determining submodule is further used to:

[0046] According to the service delay requirement, a maximum batch size batchsize that meets the service delay requirement is determined.

[0047] Optionally, the second determining submodule is further used to:

[0048] According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained;

[0049] Determine a networking method according to the total number;

[0050] The parallel mode of the large model is determined according to the total number and the form of the GPU card.

[0051] Optionally, the device may further include:

[0052] A first sending module, used to send a query request to the intelligent computing resource pool through a query interface;

[0053] The first receiving module is used to receive the GPU information sent by the intelligent computing resource pool.

[0054] Optionally, the device may further include:

[0055] A second sending module, configured to send a first message to a service requester, wherein the first message includes information on estimated GPU resources;

[0056] The second receiving module is used to receive a first response sent by the service requester and adjust the estimated GPU resources according to the first response.

[0057] Optionally, the second processing module is further used to:

[0058] If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

[0059] In a third aspect, an embodiment of the present application provides a model resource deployment device, including: a processor and a transceiver;

[0060] Wherein, the transceiver is used to obtain reasoning service requirements;

[0061] The processor is used to estimate the GPU resources required for the large model to be deployed according to the inference business requirements; deploy the required GPU resources; and when receiving an inference task, adjust the deployed GPU resources according to the inference task.

[0062] Optionally, the processor is further configured to:

[0063] The estimating the graphics processing unit (GPU) resources required for the large model to be deployed according to the inference business requirements includes:

[0064] Determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model, and the GPU information;

[0065] According to the business requirements and the GPU resources required by the model instance, estimate the GPU cluster resources corresponding to the large model.

[0066] Optionally, the model requirement includes one or more of the model scale, model structure, input token length, and output token length; the business requirement includes business latency requirement; the GPU information includes the size of the GPU graphics card;

[0067] The processor is further configured to:

[0068] Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token;

[0069] The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

[0070] Optionally, the processor is further configured to:

[0071] According to the service delay requirement, a maximum batch size batchsize that meets the service delay requirement is determined.

[0072] Optionally, the processor is further configured to:

[0073] According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained;

[0074] Determine a networking method according to the total number;

[0075] The parallel mode of the large model is determined according to the total number and the form of the GPU card.

[0076] Optionally, the transceiver is further used for:

[0077] Send query requests to the intelligent computing resource pool through the query interface;

[0078] Receive the GPU information sent by the intelligent computing resource pool.

[0079] Optionally, the transceiver is further used for:

[0080] Sending a first message to the service requester, wherein the first message includes information about estimating GPU resources;

[0081] A first response sent by the service requester is received, and the estimated GPU resources are adjusted according to the first response.

[0082] Optionally, the processor is further configured to:

[0083] If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

[0084] In a fourth aspect, an embodiment of the present application further provides a communication device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the model resource deployment method as described above are implemented.

[0085] In a fifth aspect, an embodiment of the present application further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the model resource deployment method as described above are implemented.

[0086] In a ninth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the model resource deployment method as described above.

[0087] In an embodiment of the present application, the resources required for the large model to be deployed are estimated according to the inference business needs, the estimated GPU resources are determined and the corresponding resources are deployed, and when an inference task is received, the deployed GPU resources are adjusted according to the inference task. Since the resources required for the large model are estimated, it is possible to ensure that the necessary resources are deployed in advance, and when an inference task is received, the pre-deployed GPU resources can be adjusted according to the inference task, thereby improving the utilization of GPU resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is one of the flow charts of the model resource deployment method provided in the embodiment of the present application;

[0089] Figure 2 This is the second flowchart of the model resource deployment method provided in the embodiment of the present application;

[0090] Figure 3 It is one of the structural diagrams of the model resource deployment device provided in the embodiment of the present application;

[0091] Figure 4 This is the second structural diagram of the model resource deployment device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0092] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0093] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0094] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0095] See also Figure 1 , Figure 1 is a flow chart of model resource adjustment provided by an embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0096] Step 101: Obtain reasoning business requirements.

[0097] The solution of the embodiment of the present application can be applied to a management platform or a management system. The business requirements may include the business requirements of the business request end (or user end), the model requirements of the model manager, and GPU information. The business request end is used to initiate business requests, trigger reasoning tasks, etc., and the model manager is used to manage large models, such as updates, the use of reasoning model files, etc.

[0098] For the service request end, the service requirements include service delay requirements (such as the first token delay, the remaining token delays, etc.), the number of user requests, the number of users, the user's daily active time, service accuracy requirements, etc.

[0099] The model requirements of the large model to be deployed by the model manager may include model scale, model structure, length of input tokens, length of output tokens, intermediate calculation size, etc.

[0100] The GPU information includes information related to the GPU, such as the size of the GPU graphics card, video memory bandwidth, inter-card bandwidth, video memory size, and the interconnection form between GPU cards on a single server.

[0101] For GPU information, the management platform or management system can send a query request to the intelligent computing resource pool through the query interface, and receive the GPU information sent by the intelligent computing resource pool in response to the query request. The intelligent computing resource pool is used to manage GPU resources, including GPU manufacturers, GPU supported accuracy, computing power, video memory size, etc.

[0102] Among them, the form of GPU card may include buckle card, peripheral component interconnect express (PCle), etc. Taking PCle as an example, PCle may include artificial intelligence standard card, artificial intelligence module, etc.

[0103] In the embodiment of the present application, the PCIe device set resource information can be queried through the query interface. Among them:

[0104] Operation type: GET;

[0105] URL: https: / / device_ip / redfish / v1 / Chassis / chassis_id / PCIeDevices;

[0106] Request header:

[0107] X-Auth-Token:auth_value;

[0108] Request message body: None.

[0109] The parameters for querying the PCIe device set are shown in Table 1.

[0110] Table 1

[0111]

[0112] For example, the query request may be:

[0113] GET https: / / 10.10.10.10 / redfish / v1 / Chassis / 1 / PCIeDevices; request header: X-Auth-Token:6599174c38c36838737d9749179e1ee1; request message body: none.

[0114] The intelligent computing resource pool can return the requested information to the management platform or management system in the form of a response. The information fed back by the intelligent computing resource pool is shown in Table 2.

[0115] Table 2

[0116]

[0117]

[0118] In the embodiment of the present application, the management platform or management system can also query the specified PCIe device resource information, including: PCIe device resource ID, name, description, manufacturer, model, firmware version, serial number, health status, slot number, device type, chip manufacturer, chip model, chip storage type and type, computing power description, etc. Among them:

[0119] Operation type: GET;

[0120] URL: https: / / device_ip / redfish / v1 / Chassis / chassis_id / PCIeDevices / pciedevices_id;

[0121] Request header: X-Auth-Token:auth_value;

[0122] Request message body: None.

[0123] The parameters that can be carried in the query request are shown in Table 3.

[0124] Table 3

[0125]

[0126]

[0127] The intelligent computing resource pool can return the requested information to the management platform or management system in the form of a response. The information fed back by the intelligent computing resource pool is shown in Table 4.

[0128] Table 4

[0129]

[0130]

[0131]

[0132]

[0133] In the embodiment of the present application, the management platform or management system can also query all PCIe device resource information. Among them:

[0134] Operation type: GET;

[0135] URL: https: / / device_ip / redfish / v1 / Chassis / chassis_id / PCIeDevices? $expand=.;

[0136] Request header: X-Auth-Token:auth_value;

[0137] Request message body: None.

[0138] The parameters that can be carried in the query request are shown in Table 5.

[0139] Table 5

[0140]

[0141]

[0142] The intelligent computing resource pool can return the requested information to the management platform or management system in the form of a response. The information fed back by the intelligent computing resource pool is shown in Table 6.

[0143] Table 6

[0144]

[0145] Through the above interface parameters, the management platform can effectively obtain GPU resource information to facilitate resource estimation management.

[0146] Step 102: Estimate the GPU resources required for the large model to be deployed based on the inference business requirements.

[0147] Here, based on the business requirements, the model requirements of the large model and the GPU information, the GPU resources required for a single model instance are determined, and then based on the business requirements and the GPU resources required for the model instance, the GPU cluster resources corresponding to the large model are estimated.

[0148] Specifically, when determining the GPU resources required for a single model instance, the total GPU graphics card storage required for the single model instance may be determined based on one or more of the model scale, model structure, length of input tokens, and length of output tokens. And, the number of GPU cards required for the single model instance may be determined based on the size of the GPU graphics card and the total GPU graphics card storage.

[0149] For example, the total GPU graphics card storage required for a single model instance can be estimated based on actual experience and information about the large model to be deployed. Alternatively, the total GPU graphics card storage required for a single model instance can be calculated by multiplying the model parameters of the model to be deployed by the estimated coefficients.

[0150] For example, taking the GTP-4 1800B model as an example, if the model is not quantized, 57 cards need to be deployed (taking 64GB of card memory as an example). The total GPU graphics card storage required for a single model instance is: 2.4×1800≈4320GB, where 2.4 represents the estimated coefficient and 1800 represents the model parameter quantity or model scale.

[0151] When determining the number of GPU cards required for a single model instance, the number of GPU cards required for a single model instance can be estimated based on the information of the large model to be deployed, GPU information, etc. Alternatively, the number of GPU cards required for a single model instance can be calculated using the total GPU graphics card storage required for a single model instance and the capacity of a single GPU card.

[0152] For example, taking the above example, the number of GPU cards required for a single model instance = the total GPU graphics card storage required for a single model instance / capacity of a single GPU card ≈ 4320GB / 64 ≈ 68 cards.

[0153] In the embodiment of the present application, the maximum batch size that meets the service latency requirement can also be determined according to the service latency requirement, so that the estimated GPU is more in line with the deployment requirements of the large model. For example, the maximum batch size can be estimated based on historical deployment data of the large model, or the maximum batch size can be obtained by stress testing the GPU card, so that the maximum batch size obtained is more accurate.

[0154] When estimating the GPU cluster resources corresponding to the large model, the total number of GPU cards required for the large model can be obtained based on the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size. Then, the networking method is determined based on the total number, and the parallel method of the large model is determined based on the total number and the form of the GPU cards.

[0155] The total number of GPU cards required for the large model can be calculated as follows:

[0156] Calculate the average single-card throughput (TPS), which is (maximum batch size / service latency requirement) / number of GPU cards required for a single model instance.

[0157] Calculate the total throughput rate, which is (number of users × average number of user requests) / daily active time of users.

[0158] The total number of GPU cluster resources corresponding to the large model is calculated as follows: total throughput / average single-card throughput.

[0159] Among them, the networking method includes whether there are multiple machines and whether a parameter surface is required; the card form includes PCIe, buckle card, etc.; the model parallel method includes tensor, data parallelism, etc.

[0160] Taking the above example as an example, match the latency requirements of the business with the existing models. If the latency requirement is high, use a sub-card; if the latency requirement is low, use PCIe / PCIe bridging. For the networking model segmentation method (the purpose is to solve the problem that the model cannot be placed on a single card), you can use an 8-interconnected model (TP≥8) and adopt the form of multi-machine deployment to introduce parameter plane networking.

[0161] Step 103: Deploy required GPU resources.

[0162] Here, the management platform or system can deploy the required large model to the estimated GPU cluster.

[0163] Step 104: When an inference task is received, the deployed GPU resources are adjusted according to the inference task.

[0164] Here, if the deployed GPU resources do not meet the requirements of the inference task, the deployed GPU resources are scaled up or down.

[0165] For example, if a large number of inference tasks are received and the pre-deployed GPU resources cannot meet the resources requested by the inference tasks, capacity expansion is performed; if a small number of inference tasks are received and the pre-deployed GPU resources exceed the resources required by the inference tasks, capacity reduction is performed.

[0166] In an embodiment of the present application, the resources required for the large model to be deployed are estimated according to the inference business needs, the estimated GPU resources are determined and the corresponding resources are deployed, and when an inference task is received, the deployed GPU resources are adjusted according to the inference task. Since the resources required for the large model are estimated, it is possible to ensure that the necessary resources are deployed in advance, and when an inference task is received, the pre-deployed GPU resources can be adjusted according to the inference task, thereby improving the utilization of GPU resources.

[0167] At the same time, in the embodiment of the present application, there is no need to adjust the required resources during the execution of a certain reasoning task, thereby reducing the impact on the reasoning business and improving the efficiency of reasoning.

[0168] In an embodiment of the present application, after determining the estimated GPU resources, the management platform or management system may send a first message to the service requester, wherein the first message includes information about the estimated GPU resources. Thereafter, the management platform or management system receives a first response sent by the service requester and adjusts the estimated GPU resources based on the first response.

[0169] The estimated GPU resource information may include the estimated GPU resource size, GPU card information, etc. Through the received first response, the management platform or management system can determine whether the estimated GPU resources preliminarily match the needs of the business demander, so as to deploy appropriate resources for the subsequent reasoning task processing.

[0170] See also Figure 2 , Figure 2 is a flow chart of model resource adjustment provided by an embodiment of the present application, such as Figure 2 As shown, the following steps are included:

[0171] Step 201: The management platform obtains the GPU resource status in the current intelligent computing resource pool through the query interface, including GPU manufacturer, supported accuracy, computing power, video memory size, etc.

[0172] Step 202: The business demand side and the model side send the inference business requirements to the management platform, including latency, concurrency, inference model files used by the business, whether to specify the GPU required by the business, etc.

[0173] Step 203: The management platform calculates the resources required to deploy a single instance of the model in the resource pool and the final cluster resources that meet the business requirements based on the received business requirements and the actual stress test situation.

[0174] Step 204: The management platform feeds back the final cluster resource information to the business demand side.

[0175] Step 205: The business demand side evaluates whether to accept the final cluster resource information fed back by the management platform. If accepted, a response message is sent to the management platform and step 206 is performed; if not, a response message is sent to the management platform, and the management platform can perform optimization processing such as model compression and model change. Steps 202-205 are repeated until the requirements are met.

[0176] Step 206: The business demand side sends the large model required by the business to the management platform.

[0177] Step 207: The management platform deploys the required large model to the GPU cluster determined by the management platform.

[0178] The GPU cluster deploys the model, sends a deployment completion message to the management platform, and is ready to perform inference tasks.

[0179] Step 208: The management platform completes deployment of the business feedback model and can execute reasoning tasks.

[0180] Step 209: The business sends an actual business request, ie, an inference task, to the management platform.

[0181] Step 210: The management platform evaluates whether the resources currently allocated to the business meet the request based on the received business request. If so, the business is scheduled to the cluster resources. If not, the cluster resources are scaled up or down, and steps 207-209 are repeated until the request is met.

[0182] Afterwards, the GPU cluster resources corresponding to the business execute the inference task and return the inference results to the management platform.

[0183] It can be seen from the above description that in the embodiments of the present application, large model reasoning can be evaluated in cluster resources, GPU resources can be effectively utilized, and the resource utilization rate of the intelligent computing center can be improved.

[0184] See also Figure 3 , Figure 3 is a structural diagram of the model resource deployment device provided in the embodiment of the present application. Figure 3 As shown, the model resource deployment device includes:

[0185] The first acquisition module 301 is used to obtain the inference business requirements; the first determination module 302 is used to estimate the GPU resources required for the large model to be deployed based on the inference business requirements; the first processing module 303 is used to deploy the required GPU resources; the second processing module 304 is used to adjust the deployed GPU resources according to the inference task when an inference task is received.

[0186] Optionally, the inference business requirement includes business requirements, model requirements of the large model, and GPU information; the first determination module 302 includes:

[0187] A first determination submodule is used to determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model and the GPU information;

[0188] The second determination submodule is used to estimate the GPU cluster resources corresponding to the large model according to the business requirements and the GPU resources required by the model instance.

[0189] Optionally, the model requirement includes one or more of a model scale, a model structure, a length of an input token, and a length of an output token; the service requirement includes a service latency requirement; the GPU information includes a size of a GPU graphics card; and the first determining submodule is used to:

[0190] Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token;

[0191] The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

[0192] Optionally, the first determining submodule is further used to:

[0193] According to the service delay requirement, a maximum batch size that meets the service delay requirement is determined.

[0194] Optionally, the second determining submodule is further used to:

[0195] According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained;

[0196] Determine a networking method according to the total number;

[0197] The parallel mode of the large model is determined according to the total number and the form of the GPU card.

[0198] Optionally, the device may further include:

[0199] A first sending module, used to send a query request to the intelligent computing resource pool through a query interface;

[0200] The first receiving module is used to receive the GPU information sent by the intelligent computing resource pool.

[0201] Optionally, the device may further include:

[0202] A second sending module, configured to send a first message to a service requester, wherein the first message includes information on estimated GPU resources;

[0203] The second receiving module is used to receive a first response sent by the service requester and adjust the estimated GPU resources according to the first response.

[0204] Optionally, the second processing module is further used to:

[0205] If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

[0206] The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0207] See also Figure 4 , Figure 4 is a structural diagram of the model resource deployment device provided in the embodiment of the present application. Figure 4As shown, the model resource deployment device includes: a processor 401 and a transceiver 402;

[0208] Wherein, the transceiver 402 is used to obtain the reasoning service demand;

[0209] The processor 401 is used to estimate the GPU resources required for the large model to be deployed according to the inference business requirements; deploy the required GPU resources; and when receiving an inference task, adjust the deployed GPU resources according to the inference task.

[0210] Optionally, the processor 401 is further configured to:

[0211] The estimating the graphics processing unit (GPU) resources required for the large model to be deployed according to the inference business requirements includes:

[0212] Determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model, and the GPU information;

[0213] According to the business requirements and the GPU resources required by the model instance, estimate the GPU cluster resources corresponding to the large model.

[0214] Optionally, the model requirement includes one or more of the model scale, model structure, input token length, and output token length; the business requirement includes business latency requirement; the GPU information includes the size of the GPU graphics card;

[0215] The processor 401 is further configured to:

[0216] Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token;

[0217] The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

[0218] Optionally, the processor 401 is further configured to:

[0219] According to the service delay requirement, a maximum batch size batchsize that meets the service delay requirement is determined.

[0220] Optionally, the processor 401 is further configured to:

[0221] According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained;

[0222] Determine a networking method according to the total number;

[0223] The parallel mode of the large model is determined according to the total number and the form of the GPU card.

[0224] Optionally, the transceiver 402 is further used for:

[0225] Send query requests to the intelligent computing resource pool through the query interface;

[0226] Receive the GPU information sent by the intelligent computing resource pool.

[0227] Optionally, the transceiver 402 is further used for:

[0228] Sending a first message to the service requester, wherein the first message includes information about estimating GPU resources;

[0229] A first response sent by the service requester is received, and the estimated GPU resources are adjusted according to the first response.

[0230] Optionally, the processor 401 is further configured to:

[0231] If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

[0232] The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0233] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0234] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0235] An embodiment of the present application provides a communication device, including: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the model resource deployment method described above.

[0236] The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by the processor, each process of the above-mentioned model resource deployment method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0237] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned model resource deployment method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0238] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0240] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model resource deployment method, characterized in that: include: Obtain inference business requirements; According to the inference business requirements, estimate the graphics processing unit (GPU) resources required for the large model to be deployed; Deploy the required GPU resources; When an inference task is received, the deployed GPU resources are adjusted according to the inference task.

2. The method according to claim 1, characterized in that: The inference business requirements include business requirements, model requirements of the large model, and GPU information; The estimating the graphics processing unit (GPU) resources required for the large model to be deployed according to the inference business requirements includes: Determine the GPU resources required for a single model instance according to the business requirements, the model requirements of the large model, and the GPU information; According to the business requirements and the GPU resources required by the model instance, estimate the GPU cluster resources corresponding to the large model.

3. The method according to claim 2, characterized in that The model requirements include one or more of the model scale, model structure, input token length, and output token length; the business requirements include business latency requirements; the GPU information includes the size of the GPU graphics card; The determining, according to the business requirements, the model requirements of the large model and the GPU information, the GPU resources required for the single model instance includes: Determine the total GPU graphics card storage required for a single model instance based on one or more of the model size, the model structure, the length of the input token, and the length of the output token; The number of GPU cards required for a single model instance is determined according to the size of the GPU graphics card and the total storage capacity of the GPU graphics card.

4. The method according to claim 3, characterized in that The method further comprises: According to the service delay requirement, a maximum batch size batchsize that meets the service delay requirement is determined.

5. The method according to claim 4, characterized in that The estimating the GPU cluster resources corresponding to the large model according to the business requirements and the GPU resources required by the model instance includes: According to the business requirements, the number of GPU cards required for the single model instance, and the maximum batch size, the total number of GPU cards required for the large model is obtained; Determine a networking method according to the total number; The parallel mode of the large model is determined according to the total number and the form of the GPU card.

6. The method according to claim 2, characterized in that The method further comprises: Send query requests to the intelligent computing resource pool through the query interface; Receive the GPU information sent by the intelligent computing resource pool.

7. The method according to claim 1, characterized in that The method further comprises: Sending a first message to the service requester, wherein the first message includes information about estimating GPU resources; A first response sent by the service requester is received, and the estimated GPU resources are adjusted according to the first response.

8. The method according to claim 1, characterized in that When receiving the inference task, adjusting the deployed GPU resources according to the inference task includes: If the deployed GPU resources do not meet the requirements of the reasoning task, the deployed GPU resources are scaled up or down.

9. A model resource deployment device, characterized in that: include: A first acquisition module is used to acquire reasoning business requirements; A first determination module is used to estimate the GPU resources required for the large model to be deployed according to the inference business requirements; A first processing module, used to deploy required GPU resources; The second processing module is used to adjust the deployed GPU resources according to the inference task when receiving the inference task.

10. A model resource deployment device, characterized in that: include: processors and transceivers; Wherein, the transceiver is used to obtain reasoning service requirements; The processor is used to estimate the GPU resources required for the large model to be deployed according to the inference business requirements; deploy the required GPU resources; and when receiving an inference task, adjust the deployed GPU resources according to the inference task.

11. A communication device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the model resource deployment method as described in any one of claims 1 to 8.

12. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the model resource deployment method as described in any one of claims 1 to 8 are implemented.

13. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps in the model resource deployment method as described in any one of claims 1 to 8.

Images